2021-03-02T11:57:23.860 INFO:root:teuthology version: 1.1.0-3082387b 2021-03-02T11:57:23.863 DEBUG:teuthology.run:Teuthology command: teuthology --description rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} --owner scheduled_yuriw@teuthology --name yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi --verbose --archive /home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536 -- /home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536/orig.config.yaml 2021-03-02T11:57:23.922 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-02T11:57:23.991 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536 branch: wip-yuri-testing-2021-03-01-1125-octopus description: rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind} email: ceph-qa@ceph.io first_in_suite: false job_id: '5925536' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi nuke-on-error: true os_type: ubuntu overrides: admin_socket: branch: wip-yuri-testing-2021-03-01-1125-octopus ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph global: osd heartbeat grace: 40 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon osd crush smoke test: false osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 osd fast shutdown: false rocksdb delete range threshold: 0 osd.0: osd_max_omap_entries_per_request: 10 osd.1: osd_max_omap_entries_per_request: 1000 osd.2: osd_max_omap_entries_per_request: 10000 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 3664024eebddedeee285ff2f143b16556af4e85d wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 3664024eebddedeee285ff2f143b16556af4e85d rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c1.client.1 name: test-zone3 - endpoints: - c2.client.1 is_pubsub: true name: test-zone4 workunit: branch: wip-yuri-testing-2021-03-01-1125-octopus sha1: 3664024eebddedeee285ff2f143b16556af4e85d owner: scheduled_yuriw@teuthology priority: 100 repo: https://github.com/ceph/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - c1.client.1 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 3664024eebddedeee285ff2f143b16556af4e85d sleep_before_teardown: 0 suite: rgw suite_branch: wip-yuri-testing-2021-03-01-1125-octopus suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_3664024eebddedeee285ff2f143b16556af4e85d/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 3664024eebddedeee285ff2f143b16556af4e85d targets: smithi012.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8QYmvqWVKZIAMULP+iGClqAj+x24U+MP7H3fZe4odr07HfPHg/CnVtjfdT2QyGoU5VWBGYl+SEuXm0F9MLifQPqrx89kTr4LayNySLJQrh+hjJta+XAB+16HRRmbzsRs54FclMEAO+Ogu18SUyjovXIFKv3EztPP8YYLZZ/0jwJPQUDmu1dL8y9dVg7DeVO3oVI2TxHwNoqeYdEedr35bO2P5fJZ2u11BU5j2X0gMI4J6IhdUplsGqUf2MSnMtXrod9tIteE5Qh9Kq8uBdbatCDaRYEULY9sk8PahmGsDPxGeg0gU+TKDKvpUXx2vgVkByYO4eOW0k4zylvLEhi9h smithi036.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIsrda0walO1WCtov3w5mYxsSoKKIQyKQdRRtrtL3bxQ0LNiCw/gthawcXiWIIG8Nh27MjSrXMSVKshdoXMQdf74860hF1Wrpc27wEmlEZgdly3AgAnyxTqvHcORqQLj4p7XZbiwk6uKsbXjWaEdrYGYAPt9/D6CIazdKZKRbAs/04UE4xeS/66T6cnuYPGV96mPMpyCN4FdBTvg7nOARpckzH2UkDMtOjFiT6ebPowrJeTt4qawjPfEcsG4ZeWqQPsI/xqcF3EXWoYd1Rmrobof2UL1bZ7KLD7Fl6tA6oyl1330KFYcKF9wmhCFZnqSW/BYmjtuVkDTWDwhafLP3J tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - rgw: c1.client.0: port: 8000 c1.client.1: port: 8001 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 3082387bba74fcd24c9700593d10418152d53c97 timestamp: 2021-03-01_21:06:29 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.25973 2021-03-02T11:57:23.992 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_3664024eebddedeee285ff2f143b16556af4e85d/qa; will attempt to use it 2021-03-02T11:57:23.996 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_3664024eebddedeee285ff2f143b16556af4e85d/qa/tasks 2021-03-02T11:57:23.996 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-02T11:57:23.999 INFO:teuthology.task.internal:Checking packages... 2021-03-02T11:57:24.090 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'basic' and ceph hash '3664024eebddedeee285ff2f143b16556af4e85d' 2021-03-02T11:57:24.091 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-02T11:57:24.092 INFO:teuthology.packaging:ref: None 2021-03-02T11:57:24.092 INFO:teuthology.packaging:tag: None 2021-03-02T11:57:24.092 INFO:teuthology.packaging:branch: wip-yuri-testing-2021-03-01-1125-octopus 2021-03-02T11:57:24.093 INFO:teuthology.packaging:sha1: 3664024eebddedeee285ff2f143b16556af4e85d 2021-03-02T11:57:24.093 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=wip-yuri-testing-2021-03-01-1125-octopus 2021-03-02T11:57:24.265 INFO:teuthology.task.internal:Found packages for ceph version 15.2.9-47-g3664024e-1bionic 2021-03-02T11:57:24.266 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-02T11:57:24.274 INFO:teuthology.task.internal:no buildpackages task found 2021-03-02T11:57:24.274 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-02T11:57:24.416 INFO:teuthology.task.internal:Saving configuration 2021-03-02T11:57:24.433 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-02T11:57:24.439 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-02T11:57:24.459 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-02 11:51:19.428558', 'description': '/home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA2oIkccqf9vsGfTlNqgD15QE9yF/qVtETyMyaFLYE4T', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi012.front.sepia.ceph.com'} 2021-03-02T11:57:24.475 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-02 11:51:19.428539', 'description': '/home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536', 'locked_by': 'scheduled_yuriw@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJJo2qz0Wc4u7v0QjjBqDrm5TPu51UrbExXQYr5YpBCS', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi036.front.sepia.ceph.com'} 2021-03-02T11:57:24.476 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-02T11:57:24.488 INFO:teuthology.task.internal:roles: ubuntu@smithi012.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'] 2021-03-02T11:57:24.489 INFO:teuthology.task.internal:roles: ubuntu@smithi036.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-03-02T11:57:24.489 INFO:teuthology.run_tasks:Running task console_log... 2021-03-02T11:57:24.796 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1aa634ef28>, signals=[15]) 2021-03-02T11:57:24.797 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-02T11:57:24.808 INFO:teuthology.task.internal:Opening connections... 2021-03-02T11:57:24.808 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi012.front.sepia.ceph.com 2021-03-02T11:57:24.812 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-02T11:57:24.903 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi036.front.sepia.ceph.com 2021-03-02T11:57:24.904 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi036.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-02T11:57:25.038 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-02T11:57:25.079 DEBUG:teuthology.orchestra.run.smithi012:> uname -m 2021-03-02T11:57:25.109 INFO:teuthology.orchestra.run.smithi012.stdout:x86_64 2021-03-02T11:57:25.110 DEBUG:teuthology.orchestra.run.smithi012:> cat /etc/os-release 2021-03-02T11:57:25.180 INFO:teuthology.orchestra.run.smithi012.stdout:NAME="Ubuntu" 2021-03-02T11:57:25.181 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-02T11:57:25.181 INFO:teuthology.orchestra.run.smithi012.stdout:ID=ubuntu 2021-03-02T11:57:25.181 INFO:teuthology.orchestra.run.smithi012.stdout:ID_LIKE=debian 2021-03-02T11:57:25.181 INFO:teuthology.orchestra.run.smithi012.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-02T11:57:25.181 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_ID="18.04" 2021-03-02T11:57:25.182 INFO:teuthology.orchestra.run.smithi012.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-02T11:57:25.182 INFO:teuthology.orchestra.run.smithi012.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-02T11:57:25.182 INFO:teuthology.orchestra.run.smithi012.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-02T11:57:25.182 INFO:teuthology.orchestra.run.smithi012.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-02T11:57:25.183 INFO:teuthology.orchestra.run.smithi012.stdout:VERSION_CODENAME=bionic 2021-03-02T11:57:25.183 INFO:teuthology.orchestra.run.smithi012.stdout:UBUNTU_CODENAME=bionic 2021-03-02T11:57:25.184 INFO:teuthology.lock.ops:Updating smithi012.front.sepia.ceph.com on lock server 2021-03-02T11:57:25.211 DEBUG:teuthology.orchestra.run.smithi036:> uname -m 2021-03-02T11:57:25.231 INFO:teuthology.orchestra.run.smithi036.stdout:x86_64 2021-03-02T11:57:25.232 DEBUG:teuthology.orchestra.run.smithi036:> cat /etc/os-release 2021-03-02T11:57:25.287 INFO:teuthology.orchestra.run.smithi036.stdout:NAME="Ubuntu" 2021-03-02T11:57:25.288 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-02T11:57:25.288 INFO:teuthology.orchestra.run.smithi036.stdout:ID=ubuntu 2021-03-02T11:57:25.288 INFO:teuthology.orchestra.run.smithi036.stdout:ID_LIKE=debian 2021-03-02T11:57:25.288 INFO:teuthology.orchestra.run.smithi036.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-02T11:57:25.289 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION_ID="18.04" 2021-03-02T11:57:25.289 INFO:teuthology.orchestra.run.smithi036.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-02T11:57:25.289 INFO:teuthology.orchestra.run.smithi036.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-02T11:57:25.289 INFO:teuthology.orchestra.run.smithi036.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-02T11:57:25.290 INFO:teuthology.orchestra.run.smithi036.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-02T11:57:25.290 INFO:teuthology.orchestra.run.smithi036.stdout:VERSION_CODENAME=bionic 2021-03-02T11:57:25.290 INFO:teuthology.orchestra.run.smithi036.stdout:UBUNTU_CODENAME=bionic 2021-03-02T11:57:25.290 INFO:teuthology.lock.ops:Updating smithi036.front.sepia.ceph.com on lock server 2021-03-02T11:57:25.341 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-02T11:57:25.362 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-02T11:57:25.443 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-02T11:57:25.444 DEBUG:teuthology.orchestra.run.smithi012:> test '!' -e /home/ubuntu/cephtest 2021-03-02T11:57:25.455 DEBUG:teuthology.orchestra.run.smithi036:> test '!' -e /home/ubuntu/cephtest 2021-03-02T11:57:25.466 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-02T11:57:25.559 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-02T11:57:25.560 DEBUG:teuthology.orchestra.run.smithi012:> test -z $(ls -A /var/lib/ceph) 2021-03-02T11:57:25.565 DEBUG:teuthology.orchestra.run.smithi036:> test -z $(ls -A /var/lib/ceph) 2021-03-02T11:57:25.583 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-02T11:57:25.662 INFO:teuthology.run_tasks:Running task kernel... 2021-03-02T11:57:25.691 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c1.client.1': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-03-02T11:57:25.695 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-02T11:57:25.695 DEBUG:teuthology.orchestra.run.smithi012:> uname -r 2021-03-02T11:57:25.713 INFO:teuthology.orchestra.run.smithi012.stdout:4.15.0-136-generic 2021-03-02T11:57:25.715 INFO:teuthology.task.kernel:Running kernel on smithi012: 4.15.0-136-generic 2021-03-02T11:57:25.716 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get clean 2021-03-02T11:57:25.820 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2021-03-02T11:57:26.131 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-02T11:57:26.133 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-02T11:57:26.194 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-02T11:57:26.427 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-02T11:57:26.521 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,572 kB] 2021-03-02T11:57:26.649 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-02T11:57:26.970 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [925 kB] 2021-03-02T11:57:26.990 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-03-02T11:57:27.026 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,229 kB] 2021-03-02T11:57:27.058 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-03-02T11:57:27.059 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-02T11:57:27.129 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-02T11:57:28.453 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 11.3 MB in 2s (4,769 kB/s) 2021-03-02T11:57:30.019 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2021-03-02T11:57:30.070 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-02T11:57:30.163 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2021-03-02T11:57:30.433 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2021-03-02T11:57:30.434 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2021-03-02T11:57:30.525 INFO:teuthology.orchestra.run.smithi012.stderr:E: Unable to locate package linux-image-current-generic 2021-03-02T11:57:30.526 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-02T11:57:30.527 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-02T11:57:30.609 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2021-03-02T11:57:30.851 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2021-03-02T11:57:30.852 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2021-03-02T11:57:31.070 INFO:teuthology.orchestra.run.smithi012.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-02T11:57:31.070 INFO:teuthology.orchestra.run.smithi012.stdout:linux-image-generic set to manually installed. 2021-03-02T11:57:31.070 INFO:teuthology.orchestra.run.smithi012.stdout:0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded. 2021-03-02T11:57:31.072 DEBUG:teuthology.orchestra.run.smithi012:> dpkg -s linux-image-generic 2021-03-02T11:57:31.094 INFO:teuthology.orchestra.run.smithi012.stdout:Package: linux-image-generic 2021-03-02T11:57:31.094 INFO:teuthology.orchestra.run.smithi012.stdout:Status: install ok installed 2021-03-02T11:57:31.095 INFO:teuthology.orchestra.run.smithi012.stdout:Priority: optional 2021-03-02T11:57:31.095 INFO:teuthology.orchestra.run.smithi012.stdout:Section: kernel 2021-03-02T11:57:31.096 INFO:teuthology.orchestra.run.smithi012.stdout:Installed-Size: 17 2021-03-02T11:57:31.096 INFO:teuthology.orchestra.run.smithi012.stdout:Maintainer: Ubuntu Kernel Team 2021-03-02T11:57:31.097 INFO:teuthology.orchestra.run.smithi012.stdout:Architecture: amd64 2021-03-02T11:57:31.097 INFO:teuthology.orchestra.run.smithi012.stdout:Source: linux-meta 2021-03-02T11:57:31.098 INFO:teuthology.orchestra.run.smithi012.stdout:Version: 4.15.0.136.123 2021-03-02T11:57:31.098 INFO:teuthology.orchestra.run.smithi012.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-02T11:57:31.098 INFO:teuthology.orchestra.run.smithi012.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-02T11:57:31.099 INFO:teuthology.orchestra.run.smithi012.stdout:Recommends: thermald 2021-03-02T11:57:31.099 INFO:teuthology.orchestra.run.smithi012.stdout:Description: Generic Linux kernel image 2021-03-02T11:57:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: This package will always depend on the latest generic kernel image 2021-03-02T11:57:31.100 INFO:teuthology.orchestra.run.smithi012.stdout: available. 2021-03-02T11:57:31.101 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-02T11:57:31.102 DEBUG:teuthology.orchestra.run.smithi036:> uname -r 2021-03-02T11:57:31.113 INFO:teuthology.orchestra.run.smithi036.stdout:4.15.0-136-generic 2021-03-02T11:57:31.113 INFO:teuthology.task.kernel:Running kernel on smithi036: 4.15.0-136-generic 2021-03-02T11:57:31.114 DEBUG:teuthology.orchestra.run.smithi036:> sudo apt-get clean 2021-03-02T11:57:31.234 DEBUG:teuthology.orchestra.run.smithi036:> sudo apt-get update 2021-03-02T11:57:31.683 INFO:teuthology.orchestra.run.smithi036.stdout:Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-02T11:57:31.699 INFO:teuthology.orchestra.run.smithi036.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-02T11:57:31.766 INFO:teuthology.orchestra.run.smithi036.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-02T11:57:32.074 INFO:teuthology.orchestra.run.smithi036.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-02T11:57:32.131 INFO:teuthology.orchestra.run.smithi036.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [925 kB] 2021-03-02T11:57:32.265 INFO:teuthology.orchestra.run.smithi036.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,229 kB] 2021-03-02T11:57:32.502 INFO:teuthology.orchestra.run.smithi036.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,572 kB] 2021-03-02T11:57:32.615 INFO:teuthology.orchestra.run.smithi036.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,913 kB] 2021-03-02T11:57:32.618 INFO:teuthology.orchestra.run.smithi036.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [975 kB] 2021-03-02T11:57:32.655 INFO:teuthology.orchestra.run.smithi036.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,109 kB] 2021-03-02T11:57:32.721 INFO:teuthology.orchestra.run.smithi036.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,561 kB] 2021-03-02T11:57:32.870 INFO:teuthology.orchestra.run.smithi036.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,719 kB] 2021-03-02T11:57:33.933 INFO:teuthology.orchestra.run.smithi036.stdout:Fetched 11.3 MB in 2s (4,958 kB/s) 2021-03-02T11:57:35.509 INFO:teuthology.orchestra.run.smithi036.stdout:Reading package lists... 2021-03-02T11:57:35.555 DEBUG:teuthology.orchestra.run.smithi036:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-02T11:57:35.639 INFO:teuthology.orchestra.run.smithi036.stdout:Reading package lists... 2021-03-02T11:57:35.904 INFO:teuthology.orchestra.run.smithi036.stdout:Building dependency tree... 2021-03-02T11:57:35.905 INFO:teuthology.orchestra.run.smithi036.stdout:Reading state information... 2021-03-02T11:57:35.989 INFO:teuthology.orchestra.run.smithi036.stderr:E: Unable to locate package linux-image-current-generic 2021-03-02T11:57:35.990 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-02T11:57:35.991 DEBUG:teuthology.orchestra.run.smithi036:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-02T11:57:36.133 INFO:teuthology.orchestra.run.smithi036.stdout:Reading package lists... 2021-03-02T11:57:36.398 INFO:teuthology.orchestra.run.smithi036.stdout:Building dependency tree... 2021-03-02T11:57:36.399 INFO:teuthology.orchestra.run.smithi036.stdout:Reading state information... 2021-03-02T11:57:36.618 INFO:teuthology.orchestra.run.smithi036.stdout:linux-image-generic is already the newest version (4.15.0.136.123). 2021-03-02T11:57:36.621 INFO:teuthology.orchestra.run.smithi036.stdout:linux-image-generic set to manually installed. 2021-03-02T11:57:36.622 INFO:teuthology.orchestra.run.smithi036.stdout:0 upgraded, 0 newly installed, 0 to remove and 5 not upgraded. 2021-03-02T11:57:36.622 DEBUG:teuthology.orchestra.run.smithi036:> dpkg -s linux-image-generic 2021-03-02T11:57:36.707 INFO:teuthology.orchestra.run.smithi036.stdout:Package: linux-image-generic 2021-03-02T11:57:36.707 INFO:teuthology.orchestra.run.smithi036.stdout:Status: install ok installed 2021-03-02T11:57:36.708 INFO:teuthology.orchestra.run.smithi036.stdout:Priority: optional 2021-03-02T11:57:36.708 INFO:teuthology.orchestra.run.smithi036.stdout:Section: kernel 2021-03-02T11:57:36.708 INFO:teuthology.orchestra.run.smithi036.stdout:Installed-Size: 17 2021-03-02T11:57:36.708 INFO:teuthology.orchestra.run.smithi036.stdout:Maintainer: Ubuntu Kernel Team 2021-03-02T11:57:36.708 INFO:teuthology.orchestra.run.smithi036.stdout:Architecture: amd64 2021-03-02T11:57:36.709 INFO:teuthology.orchestra.run.smithi036.stdout:Source: linux-meta 2021-03-02T11:57:36.709 INFO:teuthology.orchestra.run.smithi036.stdout:Version: 4.15.0.136.123 2021-03-02T11:57:36.709 INFO:teuthology.orchestra.run.smithi036.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-02T11:57:36.709 INFO:teuthology.orchestra.run.smithi036.stdout:Depends: linux-image-4.15.0-136-generic, linux-modules-extra-4.15.0-136-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-02T11:57:36.710 INFO:teuthology.orchestra.run.smithi036.stdout:Recommends: thermald 2021-03-02T11:57:36.710 INFO:teuthology.orchestra.run.smithi036.stdout:Description: Generic Linux kernel image 2021-03-02T11:57:36.710 INFO:teuthology.orchestra.run.smithi036.stdout: This package will always depend on the latest generic kernel image 2021-03-02T11:57:36.711 INFO:teuthology.orchestra.run.smithi036.stdout: available. 2021-03-02T11:57:36.712 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2021-03-02T11:57:36.712 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-03-02T11:57:36.712 DEBUG:teuthology.orchestra.run.smithi012:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-02T11:57:36.752 INFO:teuthology.orchestra.run.smithi012.stdout:ttyS1 2021-03-02T11:57:36.781 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-03-02T11:57:36.782 DEBUG:teuthology.orchestra.run.smithi036:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-02T11:57:36.817 INFO:teuthology.orchestra.run.smithi036.stdout:ttyS1 2021-03-02T11:57:36.848 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-02T11:57:36.860 INFO:teuthology.task.internal:Creating test directory... 2021-03-02T11:57:36.862 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-02T11:57:36.867 DEBUG:teuthology.orchestra.run.smithi036:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-02T11:57:36.900 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-02T11:57:37.119 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-02T11:57:37.363 INFO:teuthology.task.internal:Creating archive directory... 2021-03-02T11:57:37.363 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-02T11:57:37.369 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-02T11:57:37.393 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-02T11:57:37.610 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-02T11:57:37.611 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-02T11:57:37.615 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-02T11:57:37.633 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-02T11:57:37.638 INFO:teuthology.orchestra.run.smithi036.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-02T11:57:37.642 INFO:teuthology.orchestra.run.smithi012.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-02T11:57:37.648 INFO:teuthology.orchestra.run.smithi036.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-02T11:57:37.650 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-02T11:57:37.786 INFO:teuthology.task.internal:Configuring sudo... 2021-03-02T11:57:37.786 DEBUG:teuthology.orchestra.run.smithi012:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-02T11:57:37.789 DEBUG:teuthology.orchestra.run.smithi036:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-02T11:57:37.806 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-02T11:57:37.847 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-02T11:57:37.848 DEBUG:teuthology.orchestra.run.smithi012:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-02T11:57:37.852 DEBUG:teuthology.orchestra.run.smithi036:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-02T11:57:37.859 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-02T11:57:37.906 DEBUG:teuthology.orchestra.run.smithi012:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-02T11:57:37.955 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T11:57:37.956 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-02T11:57:38.024 DEBUG:teuthology.orchestra.run.smithi036:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-02T11:57:38.032 DEBUG:teuthology.orchestra.run.smithi036:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-02T11:57:38.079 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T11:57:38.080 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-02T11:57:38.142 DEBUG:teuthology.orchestra.run.smithi012:> sudo service rsyslog restart 2021-03-02T11:57:38.144 DEBUG:teuthology.orchestra.run.smithi036:> sudo service rsyslog restart 2021-03-02T11:57:38.227 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-02T11:57:38.238 INFO:teuthology.task.internal:Starting timer... 2021-03-02T11:57:38.239 INFO:teuthology.run_tasks:Running task pcp... 2021-03-02T11:57:38.271 INFO:teuthology.run_tasks:Running task selinux... 2021-03-02T11:57:38.303 DEBUG:teuthology.task.selinux:Excluding smithi012: OS 'ubuntu' does not support SELinux 2021-03-02T11:57:38.303 DEBUG:teuthology.task.selinux:Excluding smithi036: OS 'ubuntu' does not support SELinux 2021-03-02T11:57:38.304 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-02T11:57:38.304 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-02T11:57:38.304 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-02T11:57:38.305 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-02T11:57:38.345 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-02T11:57:38.346 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-02T11:57:38.585 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-02T11:57:38.589 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi012.front.sepia.ceph.com,smithi036.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-02T11:57:39.485 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-02T11:57:41.600 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-02T11:57:44.355 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-02T11:57:44.424 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:44 +0000 (0:00:00.145) 0:00:00.145 ********* 2021-03-02T11:57:44.523 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:44 +0000 (0:00:00.022) 0:00:00.168 ********* 2021-03-02T11:57:46.056 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-02T11:57:46.058 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi012.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi012.front.sepia.ceph.com] 2021-03-02T11:57:46.072 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi036.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi036.front.sepia.ceph.com] 2021-03-02T11:57:46.146 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:46 +0000 (0:00:01.699) 0:00:01.867 ********* 2021-03-02T11:57:46.183 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:46 +0000 (0:00:00.036) 0:00:01.903 ********* 2021-03-02T11:57:46.685 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-02T11:57:46.687 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-02T11:57:46.722 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:46 +0000 (0:00:00.540) 0:00:02.443 ********* 2021-03-02T11:57:46.730 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-02T11:57:46.774 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:46 +0000 (0:00:00.052) 0:00:02.495 ********* 2021-03-02T11:57:47.598 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-02T11:57:47.600 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-02T11:57:47.627 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:47 +0000 (0:00:00.852) 0:00:03.348 ********* 2021-03-02T11:57:47.638 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-02T11:57:47.671 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:47 +0000 (0:00:00.043) 0:00:03.392 ********* 2021-03-02T11:57:48.114 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-02T11:57:48.116 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.076284", "end": "2021-03-02 11:57:48.136598", "rc": 0, "start": "2021-03-02 11:57:48.060314", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.074813", "end": "2021-03-02 11:57:47.746790", "rc": 0, "start": "2021-03-02 11:57:47.671977", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-02T11:57:48.140 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:48 +0000 (0:00:00.469) 0:00:03.862 ********* 2021-03-02T11:57:48.161 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:48 +0000 (0:00:00.020) 0:00:03.882 ********* 2021-03-02T11:57:48.434 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:57:48.435 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:57:48.465 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:48 +0000 (0:00:00.303) 0:00:04.186 ********* 2021-03-02T11:57:48.499 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:48 +0000 (0:00:00.033) 0:00:04.220 ********* 2021-03-02T11:57:49.412 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686268.5387614-30817-124510496349978/source", "state": "file", "uid": 0} 2021-03-02T11:57:49.436 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:49 +0000 (0:00:00.937) 0:00:05.157 ********* 2021-03-02T11:57:49.446 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686268.5075312-30813-79010409394490/source", "state": "file", "uid": 0} 2021-03-02T11:57:49.496 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:49 +0000 (0:00:00.060) 0:00:05.217 ********* 2021-03-02T11:57:50.003 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-02T11:57:50.005 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-02T11:57:50.007 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-02T11:57:50.041 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-02T11:57:50.045 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-02T11:57:50.122 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-02T11:57:50.127 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-02T11:57:50.284 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:50 +0000 (0:00:00.787) 0:00:06.005 ********* 2021-03-02T11:57:50.304 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:50 +0000 (0:00:00.020) 0:00:06.025 ********* 2021-03-02T11:57:51.008 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-02T11:57:51.010 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] 2021-03-02T11:57:51.060 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] 2021-03-02T11:57:51.100 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-03-02T11:57:51.101 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:51 +0000 (0:00:00.795) 0:00:06.821 ********* 2021-03-02T11:57:51.179 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:51 +0000 (0:00:00.078) 0:00:06.900 ********* 2021-03-02T11:57:51.181 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:51.941 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-02T11:57:51.977 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.407234", "end": "2021-03-02 11:57:51.901856", "rc": 0, "start": "2021-03-02 11:57:51.494622", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-02T11:57:51.979 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:51 +0000 (0:00:00.800) 0:00:07.701 ********* 2021-03-02T11:57:51.994 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.408128", "end": "2021-03-02 11:57:51.903653", "rc": 0, "start": "2021-03-02 11:57:51.495525", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-02T11:57:52.036 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.056) 0:00:07.757 ********* 2021-03-02T11:57:52.167 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-02T11:57:52.173 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.009943", "end": "2021-03-02 11:57:52.200612", "failed_when_result": false, "rc": 0, "start": "2021-03-02 11:57:52.190669", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-02T11:57:52.207 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.166) 0:00:07.924 ********* 2021-03-02T11:57:52.587 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.015045", "end": "2021-03-02 11:57:51.844164", "failed_when_result": false, "rc": 0, "start": "2021-03-02 11:57:51.829119", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.046) 0:00:07.970 ********* TASK [Determine if we can skip users and pubkeys updates] ********************** ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.132) 0:00:08.103 ********* ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"perform_users_role": false}, "changed": false} Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.036) 0:00:08.140 ********* TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.091) 0:00:08.231 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.038) 0:00:08.270 ********* 2021-03-02T11:57:52.613 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-02T11:57:52.616 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:52.653 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.103) 0:00:08.374 ********* 2021-03-02T11:57:52.711 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:52.743 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.083) 0:00:08.458 ********* 2021-03-02T11:57:52.798 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-02T11:57:52.800 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:52.840 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.094) 0:00:08.552 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.092 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.034) 0:00:08.587 ********* TASK [Filter the managed_admin_users list] ************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:52 +0000 (0:00:00.135) 0:00:08.723 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:53 +0000 (0:00:00.045) 0:00:08.768 ********* TASK [users : Set UID_MIN to 1001] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.133 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:53 +0000 (0:00:00.085) 0:00:08.854 ********* 2021-03-02T11:57:53.359 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:53 +0000 (0:00:00.045) 0:00:08.899 ********* TASK [Create all admin users with sudo access.] ******************************** skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2m 2021-03-02T11:57:53.361 INFO:teuthology.task.ansible.out:w 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b037 2021-03-02T11:57:53.362 INFO:teuthology.task.ansible.out:4849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional 2021-03-02T11:57:53.364 INFO:teuthology.task.ansible.out:result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa 2021-03-02T11:57:53.366 INFO:teuthology.task.ansible.out:\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c 2021-03-02T11:57:53.367 INFO:teuthology.task.ansible.out:340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4 2021-03-02T11:57:53.369 INFO:teuthology.task.ansible.out:093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2ch 2021-03-02T11:57:53.371 INFO:teuthology.task.ansible.out:IBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansibl 2021-03-02T11:57:53.373 INFO:teuthology.task.ansible.out:e_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8 2021-03-02T11:57:53.377 INFO:teuthology.task.ansible.out:I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.380 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afe 2021-03-02T11:57:53.382 INFO:teuthology.task.ansible.out:d0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2C 2021-03-02T11:57:53.384 INFO:teuthology.task.ansible.out:n3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j 2021-03-02T11:57:53.385 INFO:teuthology.task.ansible.out:2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f 2021-03-02T11:57:53.387 INFO:teuthology.task.ansible.out:6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7 2021-03-02T11:57:53.388 INFO:teuthology.task.ansible.out:fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 2021-03-02T11:57:53.390 INFO:teuthology.task.ansible.out: 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yeh 2021-03-02T11:57:53.392 INFO:teuthology.task.ansible.out:udasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b0 2021-03-02T11:57:53.394 INFO:teuthology.task.ansible.out:9d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN 2021-03-02T11:57:53.395 INFO:teuthology.task.ansible.out:1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.s 2021-03-02T11:57:53.397 INFO:teuthology.task.ansible.out:epia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.705 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wus 2021-03-02T11:57:53.706 INFO:teuthology.task.ansible.out:ui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": 2021-03-02T11:57:53.708 INFO:teuthology.task.ansible.out:"item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7 2021-03-02T11:57:53.709 INFO:teuthology.task.ansible.out:c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yeh 2021-03-02T11:57:53.711 INFO:teuthology.task.ansible.out:udasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item" 2021-03-02T11:57:53.713 INFO:teuthology.task.ansible.out:: {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => 2021-03-02T11:57:53.714 INFO:teuthology.task.ansible.out: {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide 2021-03-02T11:57:53.716 INFO:teuthology.task.ansible.out: uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => 2021-03-02T11:57:53.718 INFO:teuthology.task.ansible.out: {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": " 2021-03-02T11:57:53.718 INFO:teuthology.task.ansible.out:jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.720 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.720 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.721 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.722 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.724 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.725 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.726 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.726 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.728 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.729 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.730 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.730 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmoha 2021-03-02T11:57:53.732 INFO:teuthology.task.ansible.out:n@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.732 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.734 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.735 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.737 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.738 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.739 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.739 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.741 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.742 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.744 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.746 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.748 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.748 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.748 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.750 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.750 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.751 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.752 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.753 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.761 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.762 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.764 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.767 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.770 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {" 2021-03-02T11:57:53.772 INFO:teuthology.task.ansible.out:ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.777 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.780 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud 2021-03-02T11:57:53.782 INFO:teuthology.task.ansible.out:1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.782 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.783 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451 2021-03-02T11:57:53.785 INFO:teuthology.task.ansible.out:066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b778384 2021-03-02T11:57:53.787 INFO:teuthology.task.ansible.out:59ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.788 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.789 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item" 2021-03-02T11:57:53.791 INFO:teuthology.task.ansible.out:: {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.793 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.794 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@ 2021-03-02T11:57:53.796 INFO:teuthology.task.ansible.out:p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a632 2021-03-02T11:57:53.800 INFO:teuthology.task.ansible.out:6b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var" 2021-03-02T11:57:53.802 INFO:teuthology.task.ansible.out:: "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} ski 2021-03-02T11:57:53.804 INFO:teuthology.task.ansible.out:pping: [smithi036.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"nam 2021-03-02T11:57:53.806 INFO:teuthology.task.ansible.out:e": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com 2021-03-02T11:57:53.806 INFO:teuthology.task.ansible.out:] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.807 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b 2021-03-02T11:57:53.810 INFO:teuthology.task.ansible.out:9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab 2021-03-02T11:57:53.812 INFO:teuthology.task.ansible.out:7837f"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed 2021-03-02T11:57:53.814 INFO:teuthology.task.ansible.out:": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.815 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.815 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.817 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.819 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.821 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.821 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.821 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.823 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.824 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.824 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.825 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", 2021-03-02T11:57:53.826 INFO:teuthology.task.ansible.out:"ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.836 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.837 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.838 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.839 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.839 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.851 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.852 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.854 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.854 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.856 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.857 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.857 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.869 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.870 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.871 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.881 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.883 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.895 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.897 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.908 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.909 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.920 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.922 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.924 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.925 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.936 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.938 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.939 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.950 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.953 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3d 2021-03-02T11:57:53.954 INFO:teuthology.task.ansible.out:a3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.964 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.966 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.977 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.979 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.979 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "a 2021-03-02T11:57:53.981 INFO:teuthology.task.ansible.out:dking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.991 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:53.993 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.032 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.224 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.895) 0:00:09.795 ********* Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.021) 0:00:09.817 ********* TASK [Create all users without sudo access.] *********************************** Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.077) 0:00:09.895 ********* Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.037) 0:00:09.933 ********* 2021-03-02T11:57:54.253 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-02T11:57:54.255 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.280 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.068) 0:00:10.001 ********* 2021-03-02T11:57:54.304 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.337 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.056) 0:00:10.058 ********* 2021-03-02T11:57:54.367 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-02T11:57:54.505 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.055) 0:00:10.113 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.059) 0:00:10.172 ********* TASK [users : Update authorized_keys using the keys repo] ********************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.508 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.057) 0:00:10.230 ********* 2021-03-02T11:57:54.526 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.554 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.044) 0:00:10.274 ********* 2021-03-02T11:57:54.571 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-02T11:57:54.572 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.599 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.046) 0:00:10.320 ********* 2021-03-02T11:57:54.628 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.653 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.053) 0:00:10.374 ********* 2021-03-02T11:57:54.680 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-02T11:57:54.682 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.706 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.052) 0:00:10.427 ********* 2021-03-02T11:57:54.928 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:54 +0000 (0:00:00.048) 0:00:10.475 ********* TASK [Remove revoked users] **************************************************** skipping: [smithi012.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhod 2021-03-02T11:57:54.930 INFO:teuthology.task.ansible.out:en", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was F 2021-03-02T11:57:54.932 INFO:teuthology.task.ansible.out:alse"} skipping: [smithi036.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com 2021-03-02T11:57:54.934 INFO:teuthology.task.ansible.out:] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=karnan) => {"ansible_lo 2021-03-02T11:57:54.936 INFO:teuthology.task.ansible.out:op_var": "item", "changed": false, "item": "karnan", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.941 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.943 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.943 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.943 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.944 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.944 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.946 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.956 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.958 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.959 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.959 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.970 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.972 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.982 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.984 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.986 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.997 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:54.998 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.000 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.032 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:55 +0000 (0:00:00.277) 0:00:10.753 ********* 2021-03-02T11:57:55.040 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.042 INFO:teuthology.task.ansible.out: skipping: [smithi036.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.044 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.044 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.045 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.045 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.046 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.047 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "skip_reason": "Conditional result was False"} 2021-03-02T11:57:55.077 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:55 +0000 (0:00:00.045) 0:00:10.798 ********* 2021-03-02T11:57:55.754 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2021-03-02T11:57:55.795 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:55 +0000 (0:00:00.718) 0:00:11.516 ********* 2021-03-02T11:57:55.806 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/keys-repo-sha1", "size": 40, "state": "file", "uid": 0} 2021-03-02T11:57:55.843 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:55 +0000 (0:00:00.047) 0:00:11.564 ********* 2021-03-02T11:57:55.881 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-02T11:57:55.883 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => { "msg": "Host smithi012.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-02T11:57:55.908 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:55 +0000 (0:00:00.064) 0:00:11.629 ********* 2021-03-02T11:57:55.927 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => { "msg": "Host smithi036.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-02T11:57:55.955 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:55 +0000 (0:00:00.047) 0:00:11.676 ********* 2021-03-02T11:57:55.996 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-02T11:57:55.998 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-02T11:57:56.024 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.068) 0:00:11.745 ********* 2021-03-02T11:57:56.042 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-02T11:57:56.220 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.069) 0:00:11.814 ********* TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.090) 0:00:11.905 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.228 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.044) 0:00:11.949 ********* 2021-03-02T11:57:56.267 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-02T11:57:56.268 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.292 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.063) 0:00:12.013 ********* 2021-03-02T11:57:56.342 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.367 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.075) 0:00:12.088 ********* 2021-03-02T11:57:56.386 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-02T11:57:56.560 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.045) 0:00:12.134 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.053) 0:00:12.187 ********* TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.068) 0:00:12.255 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.579 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.044) 0:00:12.300 ********* 2021-03-02T11:57:56.610 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-02T11:57:56.611 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.640 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.060) 0:00:12.361 ********* 2021-03-02T11:57:56.661 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.703 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.063) 0:00:12.424 ********* 2021-03-02T11:57:56.908 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.049) 0:00:12.474 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.077) 0:00:12.552 ********* TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.039) 0:00:12.591 ********* 2021-03-02T11:57:56.909 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.934 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.064) 0:00:12.655 ********* 2021-03-02T11:57:56.951 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-02T11:57:56.953 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:56.977 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:56 +0000 (0:00:00.043) 0:00:12.698 ********* 2021-03-02T11:57:57.022 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.064 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.086) 0:00:12.785 ********* 2021-03-02T11:57:57.144 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:57.146 INFO:teuthology.task.ansible.out:TASK [common : Include Red Hat encrypted variables.] *************************** skipping: [smithi012.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:57.147 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:57.172 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.108) 0:00:12.893 ********* 2021-03-02T11:57:57.244 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:57.246 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:57.271 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.098) 0:00:12.992 ********* 2021-03-02T11:57:57.294 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-02T11:57:57.296 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.340 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.069) 0:00:13.061 ********* 2021-03-02T11:57:57.404 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.444 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.092) 0:00:13.165 ********* 2021-03-02T11:57:57.526 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.553 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.109) 0:00:13.274 ********* 2021-03-02T11:57:57.583 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.621 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.067) 0:00:13.342 ********* 2021-03-02T11:57:57.652 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:57.654 INFO:teuthology.task.ansible.out:TASK [common : Determine if node is registered with subscription-manager.] ***** skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:57.689 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.067) 0:00:13.410 ********* 2021-03-02T11:57:57.766 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:57.768 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:57.804 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.115) 0:00:13.525 ********* 2021-03-02T11:57:57.811 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-02T11:57:57.813 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.851 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.046) 0:00:13.572 ********* 2021-03-02T11:57:57.918 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:57.949 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:57 +0000 (0:00:00.097) 0:00:13.670 ********* 2021-03-02T11:57:57.990 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:57.992 INFO:teuthology.task.ansible.out:TASK [common : Register with subscription-manager.] **************************** skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:58.018 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.069) 0:00:13.739 ********* 2021-03-02T11:57:58.068 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:58.098 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.080) 0:00:13.819 ********* 2021-03-02T11:57:58.160 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.187 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.089) 0:00:13.908 ********* 2021-03-02T11:57:58.223 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.248 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.061) 0:00:13.970 ********* 2021-03-02T11:57:58.297 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-02T11:57:58.299 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.325 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.076) 0:00:14.046 ********* 2021-03-02T11:57:58.355 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.380 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.054) 0:00:14.101 ********* 2021-03-02T11:57:58.409 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.443 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.063) 0:00:14.164 ********* 2021-03-02T11:57:58.483 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.509 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.065) 0:00:14.229 ********* 2021-03-02T11:57:58.537 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-02T11:57:58.539 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.568 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.059) 0:00:14.289 ********* 2021-03-02T11:57:58.640 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:58.667 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.098) 0:00:14.388 ********* 2021-03-02T11:57:58.668 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:58.675 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:58.676 INFO:teuthology.task.ansible.out:TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:58.721 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.054) 0:00:14.442 ********* 2021-03-02T11:57:58.750 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:58.776 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.055) 0:00:14.497 ********* 2021-03-02T11:57:58.979 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-02T11:57:58.981 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.004 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:58 +0000 (0:00:00.227) 0:00:14.725 ********* 2021-03-02T11:57:59.011 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.037 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.032) 0:00:14.758 ********* 2021-03-02T11:57:59.215 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:59.217 INFO:teuthology.task.ansible.out:TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.241 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.204) 0:00:14.962 ********* 2021-03-02T11:57:59.271 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.298 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.056) 0:00:15.019 ********* 2021-03-02T11:57:59.316 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-02T11:57:59.318 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.348 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.049) 0:00:15.069 ********* 2021-03-02T11:57:59.377 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.403 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.055) 0:00:15.124 ********* 2021-03-02T11:57:59.405 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:59.422 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-02T11:57:59.424 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.451 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.047) 0:00:15.172 ********* 2021-03-02T11:57:59.468 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.495 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.043) 0:00:15.216 ********* 2021-03-02T11:57:59.524 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-02T11:57:59.526 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.549 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.054) 0:00:15.270 ********* 2021-03-02T11:57:59.567 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.593 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.043) 0:00:15.314 ********* 2021-03-02T11:57:59.631 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-02T11:57:59.632 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.662 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.069) 0:00:15.383 ********* 2021-03-02T11:57:59.664 INFO:teuthology.task.ansible.out: 2021-03-02T11:57:59.671 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:57:59.699 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.036) 0:00:15.420 ********* 2021-03-02T11:57:59.906 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-02T11:57:59.908 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:57:59.932 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.233) 0:00:15.653 ********* 2021-03-02T11:57:59.956 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:57:59 +0000 (0:00:00.024) 0:00:15.677 ********* 2021-03-02T11:58:00.201 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:00.205 INFO:teuthology.task.ansible.out:TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-02T11:58:00.207 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.230 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.273) 0:00:15.951 ********* 2021-03-02T11:58:00.259 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.028) 0:00:15.980 ********* 2021-03-02T11:58:00.464 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-02T11:58:00.466 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.595 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.336) 0:00:16.316 ********* 2021-03-02T11:58:00.606 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.637 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.041) 0:00:16.358 ********* 2021-03-02T11:58:00.715 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.740 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.103) 0:00:16.461 ********* 2021-03-02T11:58:00.749 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.778 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.037) 0:00:16.499 ********* 2021-03-02T11:58:00.855 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-02T11:58:00.857 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.895 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:00 +0000 (0:00:00.116) 0:00:16.615 ********* 2021-03-02T11:58:00.966 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:00.967 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.016 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.122) 0:00:16.737 ********* 2021-03-02T11:58:01.036 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-02T11:58:01.038 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.070 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.046) 0:00:16.784 ********* 2021-03-02T11:58:01.195 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.197 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.227 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.163) 0:00:16.948 ********* 2021-03-02T11:58:01.245 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-02T11:58:01.246 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.275 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.048) 0:00:16.996 ********* 2021-03-02T11:58:01.315 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.342 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.066) 0:00:17.063 ********* 2021-03-02T11:58:01.507 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-02T11:58:01.508 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012816", "end": "2021-03-02 11:58:01.527539", "rc": 0, "start": "2021-03-02 11:58:01.514723", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-02T11:58:01.541 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.199) 0:00:17.262 ********* 2021-03-02T11:58:01.561 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.012837", "end": "2021-03-02 11:58:01.201555", "rc": 0, "start": "2021-03-02 11:58:01.188718", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-02T11:58:01.587 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.045) 0:00:17.308 ********* 2021-03-02T11:58:01.617 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:01.619 INFO:teuthology.task.ansible.out:TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.651 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.063) 0:00:17.372 ********* 2021-03-02T11:58:01.682 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.714 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.063) 0:00:17.435 ********* 2021-03-02T11:58:01.732 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-02T11:58:01.735 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.774 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.059) 0:00:17.495 ********* 2021-03-02T11:58:01.855 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.884 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.108) 0:00:17.605 ********* 2021-03-02T11:58:01.916 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-02T11:58:01.918 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.918 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:01.946 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:01 +0000 (0:00:00.061) 0:00:17.667 ********* 2021-03-02T11:58:02.006 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.008 INFO:teuthology.task.ansible.out: skipping: [smithi036.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.018 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.019 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:02.051 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.104) 0:00:17.772 ********* 2021-03-02T11:58:02.068 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-02T11:58:02.070 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.097 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.046) 0:00:17.818 ********* 2021-03-02T11:58:02.147 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.175 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.077) 0:00:17.896 ********* 2021-03-02T11:58:02.213 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:02.214 INFO:teuthology.task.ansible.out:TASK [common : Set the timezone] *********************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.241 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.066) 0:00:17.962 ********* 2021-03-02T11:58:02.243 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:02.264 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.301 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.059) 0:00:18.022 ********* 2021-03-02T11:58:02.332 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-02T11:58:02.333 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.361 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.059) 0:00:18.082 ********* 2021-03-02T11:58:02.394 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.433 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.071) 0:00:18.154 ********* 2021-03-02T11:58:02.465 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-02T11:58:02.467 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.511 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.077) 0:00:18.232 ********* 2021-03-02T11:58:02.567 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:02.593 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.082) 0:00:18.315 ********* 2021-03-02T11:58:02.754 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.083) 0:00:18.398 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:02 +0000 (0:00:00.073) 0:00:18.472 ********* 2021-03-02T11:58:03.477 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-02T11:58:03.478 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.636066", "end": "2021-03-02 11:58:03.500332", "rc": 0, "start": "2021-03-02 11:58:02.864266", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:03.505 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:03 +0000 (0:00:00.753) 0:00:19.226 ********* 2021-03-02T11:58:03.575 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-02T11:58:03.577 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:03.600 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:03 +0000 (0:00:00.095) 0:00:19.321 ********* 2021-03-02T11:58:03.850 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-02T11:58:03.853 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.963765", "end": "2021-03-02 11:58:03.500335", "rc": 0, "start": "2021-03-02 11:58:02.536570", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:03.876 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:03 +0000 (0:00:00.275) 0:00:19.597 ********* 2021-03-02T11:58:03.968 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-02T11:58:03.970 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:03.995 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:03 +0000 (0:00:00.118) 0:00:19.716 ********* 2021-03-02T11:58:04.222 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-02T11:58:04.224 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.023306", "end": "2021-03-02 11:58:03.861006", "rc": 0, "start": "2021-03-02 11:58:03.837700", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:04.248 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:04 +0000 (0:00:00.253) 0:00:19.969 ********* 2021-03-02T11:58:04.413 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.612478", "end": "2021-03-02 11:58:04.435350", "rc": 0, "start": "2021-03-02 11:58:03.822872", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:04.461 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:04 +0000 (0:00:00.192) 0:00:20.161 ********* 2021-03-02T11:58:07.567 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-02T11:58:07.569 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:07.592 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:07 +0000 (0:00:03.151) 0:00:23.313 ********* 2021-03-02T11:58:07.620 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:07.648 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:07 +0000 (0:00:00.055) 0:00:23.369 ********* 2021-03-02T11:58:08.644 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-02T11:58:08.647 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:08.670 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:08 +0000 (0:00:01.022) 0:00:24.391 ********* 2021-03-02T11:58:08.709 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:08.735 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:08 +0000 (0:00:00.065) 0:00:24.456 ********* 2021-03-02T11:58:08.755 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:08.801 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:08 +0000 (0:00:00.065) 0:00:24.522 ********* 2021-03-02T11:58:08.826 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:08.852 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:08 +0000 (0:00:00.051) 0:00:24.573 ********* 2021-03-02T11:58:09.294 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-02T11:58:09.296 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-02T11:58:09.324 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:09 +0000 (0:00:00.471) 0:00:25.045 ********* 2021-03-02T11:58:09.331 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-02T11:58:09.337 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:09.359 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:09 +0000 (0:00:00.035) 0:00:25.080 ********* 2021-03-02T11:58:10.285 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-02T11:58:10.286 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:10.312 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:10 +0000 (0:00:00.952) 0:00:26.033 ********* 2021-03-02T11:58:10.467 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:10.510 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:10 +0000 (0:00:00.192) 0:00:26.225 ********* 2021-03-02T11:58:10.779 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-02T11:58:10.780 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-02T11:58:11.037 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-02T11:58:11.121 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-02T11:58:11.351 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-02T11:58:11.426 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-02T11:58:11.452 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:11 +0000 (0:00:00.947) 0:00:27.173 ********* 2021-03-02T11:58:11.584 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-02T11:58:11.586 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:58:11.610 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:11 +0000 (0:00:00.157) 0:00:27.331 ********* 2021-03-02T11:58:11.650 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-02T11:58:11.652 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-02T11:58:11.678 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:11 +0000 (0:00:00.067) 0:00:27.399 ********* 2021-03-02T11:58:11.833 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:58:11.859 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:11 +0000 (0:00:00.181) 0:00:27.580 ********* 2021-03-02T11:58:11.982 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-02T11:58:11.984 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-02T11:58:12.248 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-02T11:58:12.280 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-02T11:58:12.544 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-02T11:58:12.577 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-02T11:58:12.604 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:12 +0000 (0:00:00.745) 0:00:28.325 ********* 2021-03-02T11:58:12.685 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-02T11:58:12.687 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-02T11:58:12.720 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:12 +0000 (0:00:00.115) 0:00:28.441 ********* 2021-03-02T11:58:12.802 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-02T11:58:12.803 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:12.840 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:12 +0000 (0:00:00.119) 0:00:28.560 ********* 2021-03-02T11:58:12.862 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-02T11:58:12.864 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-02T11:58:12.894 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:12 +0000 (0:00:00.054) 0:00:28.615 ********* 2021-03-02T11:58:12.936 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-02T11:58:12.937 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:58:12.966 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:12 +0000 (0:00:00.071) 0:00:28.687 ********* 2021-03-02T11:58:12.987 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-02T11:58:12.989 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-02T11:58:13.024 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.058) 0:00:28.745 ********* 2021-03-02T11:58:13.052 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-02T11:58:13.055 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.064 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.065 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.066 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.092 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.068) 0:00:28.813 ********* 2021-03-02T11:58:13.119 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-02T11:58:13.121 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.146 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.054) 0:00:28.867 ********* 2021-03-02T11:58:13.163 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-02T11:58:13.165 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.197 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.051) 0:00:28.919 ********* 2021-03-02T11:58:13.224 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-02T11:58:13.226 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:58:13.250 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.051) 0:00:28.971 ********* 2021-03-02T11:58:13.344 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-02T11:58:13.346 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.356 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.358 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:13.380 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.130) 0:00:29.101 ********* 2021-03-02T11:58:13.471 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-02T11:58:13.675 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:13 +0000 (0:00:00.113) 0:00:29.215 ********* 2021-03-02T11:58:14.175 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-02T11:58:14.570 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} Tuesday 02 March 2021 11:58:14 +0000 (0:00:00.704) 0:00:29.920 ********* ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} Tuesday 02 March 2021 11:58:14 +0000 (0:00:00.328) 0:00:30.248 ********* 2021-03-02T11:58:14.608 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-02T11:58:14.609 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-02T11:58:14.637 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:14 +0000 (0:00:00.110) 0:00:30.358 ********* 2021-03-02T11:58:14.925 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-02T11:58:15.105 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:14 +0000 (0:00:00.324) 0:00:30.683 ********* TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.077) 0:00:30.761 ********* 2021-03-02T11:58:15.268 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-02T11:58:15.452 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.326) 0:00:31.088 ********* TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ ok: [smithi036.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.044) 0:00:31.132 ********* 2021-03-02T11:58:15.491 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-02T11:58:15.493 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:15.516 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.104) 0:00:31.237 ********* 2021-03-02T11:58:15.565 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-02T11:58:15.567 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-02T11:58:15.598 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-02T11:58:15.671 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.117) 0:00:31.354 ********* 2021-03-02T11:58:15.696 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:15.697 INFO:teuthology.task.ansible.out:TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-02T11:58:15.699 INFO:teuthology.task.ansible.out: ok: [smithi012.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-02T11:58:15.723 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.090) 0:00:31.444 ********* 2021-03-02T11:58:15.730 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-02T11:58:15.732 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:15.756 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.032) 0:00:31.477 ********* 2021-03-02T11:58:15.794 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-02T11:58:15.796 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:15.819 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.062) 0:00:31.540 ********* 2021-03-02T11:58:15.889 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-02T11:58:15.891 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-02T11:58:15.913 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:15 +0000 (0:00:00.093) 0:00:31.634 ********* 2021-03-02T11:58:16.126 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:16 +0000 (0:00:00.135) 0:00:31.769 ********* 2021-03-02T11:58:16.140 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-02T11:58:16.142 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-02T11:58:16.173 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:16 +0000 (0:00:00.124) 0:00:31.894 ********* 2021-03-02T11:58:16.451 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-02T11:58:16.453 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:16.476 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:16 +0000 (0:00:00.303) 0:00:32.197 ********* 2021-03-02T11:58:16.909 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-02T11:58:16.911 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-02 11:56:53 UTC", "ActiveEnterTimestampMonotonic": "17697249", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target system.slice -.mount nss-lookup.target basic.target network.target systemd-tmpfiles-setup.service var-run.mount sysinit.target systemd-journald.socket time-sync.target local-fs.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-02 11:56:53 UTC", "AssertTimestampMonotonic": "17695905", "Before": "shutdown.target getty@tty1.service xdm.service multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-02 11:56:53 UTC", "ConditionTimestampMonotonic": "17695905", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-02T11:58:16.913 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1361", "ExecMainStartTimestamp": "Tue 2021-03-02 11:56:53 UTC", "ExecMainStartTimestampMonotonic": "17697214", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-02 11:56:53 UTC", "InactiveExitTimestampMonotonic": "17697249", "InvocationID": "485b50e23d3c4c5aa0b34c9adb56dbfa", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-02T11:58:16.914 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1361", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-02T11:58:16.916 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:56:53 UTC", "StateChangeTimestampMonotonic": "17697249", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-02 11:56:53 UTC", "WatchdogTimestampMonotonic": "17697248", "WatchdogUSec": "0"}} 2021-03-02T11:58:16.918 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-02 11:55:39 UTC", "ActiveEnterTimestampMonotonic": "17429258", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "systemd-tmpfiles-setup.service basic.target system.slice systemd-journald.socket nss-lookup.target time-sync.target remote-fs.target -.mount var-run.mount local-fs.target network.target sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-02 11:55:39 UTC", "AssertTimestampMonotonic": "17428534", "Before": "xdm.service getty@tty1.service shutdown.target multi-user.target plymouth-quit.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-02 11:55:39 UTC", "ConditionTimestampMonotonic": "17428534", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-02T11:58:16.920 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1496", "ExecMainStartTimestamp": "Tue 2021-03-02 11:55:39 UTC", "ExecMainStartTimestampMonotonic": "17429232", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-02 11:55:39 UTC", "InactiveExitTimestampMonotonic": "17429258", "InvocationID": "10d845bee2314611a1b3d4d0fc34c243", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "i 2021-03-02T11:58:16.922 INFO:teuthology.task.ansible.out:nfinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1496", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no" 2021-03-02T11:58:16.924 INFO:teuthology.task.ansible.out:, "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:55:39 UTC", "StateChangeTimestampMonotonic": "17429258", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-02 11:55:39 UTC", "WatchdogTimestampMonotonic": "17429257", "WatchdogUSec": "0"}} 2021-03-02T11:58:16.947 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:16 +0000 (0:00:00.470) 0:00:32.668 ********* 2021-03-02T11:58:16.978 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:16 +0000 (0:00:00.030) 0:00:32.699 ********* 2021-03-02T11:58:17.056 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.076) 0:00:32.776 ********* 2021-03-02T11:58:17.093 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.118 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.063) 0:00:32.839 ********* 2021-03-02T11:58:17.135 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-02T11:58:17.137 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.160 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.042) 0:00:32.881 ********* 2021-03-02T11:58:17.188 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.214 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.053) 0:00:32.935 ********* 2021-03-02T11:58:17.231 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-02T11:58:17.233 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.257 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.042) 0:00:32.978 ********* 2021-03-02T11:58:17.296 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.321 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.063) 0:00:33.042 ********* 2021-03-02T11:58:17.376 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.052) 0:00:33.094 ********* 2021-03-02T11:58:17.411 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.440 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.067) 0:00:33.161 ********* 2021-03-02T11:58:17.442 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:17.462 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-02T11:58:17.463 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.488 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.048) 0:00:33.209 ********* 2021-03-02T11:58:17.527 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.553 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.064) 0:00:33.274 ********* 2021-03-02T11:58:17.582 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-02T11:58:17.583 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.617 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.063) 0:00:33.338 ********* 2021-03-02T11:58:17.634 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.668 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.051) 0:00:33.389 ********* 2021-03-02T11:58:17.710 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-02T11:58:17.712 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.738 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.069) 0:00:33.458 ********* 2021-03-02T11:58:17.755 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.781 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.043) 0:00:33.502 ********* 2021-03-02T11:58:17.809 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-02T11:58:17.810 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.835 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.054) 0:00:33.556 ********* 2021-03-02T11:58:17.852 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.877 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.042) 0:00:33.599 ********* 2021-03-02T11:58:17.916 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-02T11:58:17.918 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.942 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.063) 0:00:33.663 ********* 2021-03-02T11:58:17.960 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:17.987 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:17 +0000 (0:00:00.045) 0:00:33.708 ********* 2021-03-02T11:58:18.017 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-02T11:58:18.135 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.056) 0:00:33.765 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.061) 0:00:33.827 ********* TASK [testnode : Include package type specific vars.] ************************** ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-02T11:58:18.151 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.045) 0:00:33.872 ********* 2021-03-02T11:58:18.193 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-02T11:58:18.219 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.068) 0:00:33.940 ********* 2021-03-02T11:58:18.251 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:18.253 INFO:teuthology.task.ansible.out:TASK [testnode : Including distro specific variables.] ************************* ok: [smithi012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-02T11:58:18.285 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.065) 0:00:34.006 ********* 2021-03-02T11:58:18.330 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "collectl", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-02T11:58:18.359 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.073) 0:00:34.080 ********* 2021-03-02T11:58:18.397 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-02T11:58:18.399 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-02T11:58:18.425 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.066) 0:00:34.146 ********* 2021-03-02T11:58:18.465 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-02T11:58:18.492 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.066) 0:00:34.212 ********* 2021-03-02T11:58:18.532 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-02T11:58:18.535 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-02T11:58:18.559 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.067) 0:00:34.280 ********* 2021-03-02T11:58:18.609 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-02T11:58:18.636 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.076) 0:00:34.356 ********* 2021-03-02T11:58:18.716 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-02T11:58:18.718 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-02T11:58:18.742 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.106) 0:00:34.463 ********* 2021-03-02T11:58:18.782 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-02T11:58:18.814 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.071) 0:00:34.535 ********* 2021-03-02T11:58:18.972 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-02T11:58:18.998 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:18 +0000 (0:00:00.183) 0:00:34.719 ********* 2021-03-02T11:58:19.029 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-02T11:58:19.054 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.056) 0:00:34.775 ********* 2021-03-02T11:58:19.072 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-02T11:58:19.074 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:19.097 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.042) 0:00:34.818 ********* 2021-03-02T11:58:19.136 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:19.162 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.065) 0:00:34.883 ********* 2021-03-02T11:58:19.199 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-02T11:58:19.204 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:19.232 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.070) 0:00:34.953 ********* 2021-03-02T11:58:19.250 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:19.277 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.044) 0:00:34.998 ********* 2021-03-02T11:58:19.462 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-02T11:58:19.465 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-02T11:58:19.488 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.211) 0:00:35.209 ********* 2021-03-02T11:58:19.506 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-02T11:58:19.531 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:19 +0000 (0:00:00.042) 0:00:35.252 ********* 2021-03-02T11:58:19.999 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-02T11:58:20.001 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-02T11:58:20.026 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:20 +0000 (0:00:00.494) 0:00:35.747 ********* 2021-03-02T11:58:20.086 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-02T11:58:20.111 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:20 +0000 (0:00:00.085) 0:00:35.832 ********* 2021-03-02T11:58:20.118 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-02T11:58:20.119 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi012"}, "changed": false} 2021-03-02T11:58:20.144 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:20 +0000 (0:00:00.033) 0:00:35.865 ********* 2021-03-02T11:58:20.183 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi036"}, "changed": false} 2021-03-02T11:58:20.207 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:20 +0000 (0:00:00.063) 0:00:35.929 ********* 2021-03-02T11:58:21.043 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-02T11:58:21.048 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi036", "ansible_hostname": "smithi036", "ansible_nodename": "smithi036"}, "changed": false, "name": "smithi036"} 2021-03-02T11:58:21.074 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:21 +0000 (0:00:00.866) 0:00:36.795 ********* 2021-03-02T11:58:21.082 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "", "ansible_fqdn": "smithi012", "ansible_hostname": "smithi012", "ansible_nodename": "smithi012"}, "changed": false, "name": "smithi012"} 2021-03-02T11:58:21.118 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:21 +0000 (0:00:00.043) 0:00:36.839 ********* 2021-03-02T11:58:21.510 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-02T11:58:21.535 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:21 +0000 (0:00:00.417) 0:00:37.256 ********* 2021-03-02T11:58:21.544 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-02T11:58:21.584 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:21 +0000 (0:00:00.049) 0:00:37.305 ********* 2021-03-02T11:58:22.000 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-02T11:58:22.002 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-02T11:58:22.026 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.441) 0:00:37.747 ********* 2021-03-02T11:58:22.052 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.025) 0:00:37.773 ********* 2021-03-02T11:58:22.529 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-02T11:58:22.531 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} ok: [smithi012.front.sepia.ceph.com] => {"attempts": 1, "changed": false, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-02T11:58:22.555 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.503) 0:00:38.276 ********* 2021-03-02T11:58:22.578 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.023) 0:00:38.299 ********* 2021-03-02T11:58:22.650 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-02T11:58:22.652 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:22.675 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.096) 0:00:38.396 ********* 2021-03-02T11:58:22.699 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.023) 0:00:38.420 ********* 2021-03-02T11:58:22.752 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-02T11:58:22.753 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:22.781 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.082) 0:00:38.502 ********* 2021-03-02T11:58:22.789 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:22.817 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.035) 0:00:38.538 ********* 2021-03-02T11:58:22.845 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-02T11:58:22.847 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:22.871 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.053) 0:00:38.592 ********* 2021-03-02T11:58:22.920 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:22.945 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.074) 0:00:38.666 ********* 2021-03-02T11:58:22.954 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-02T11:58:22.955 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:22.983 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:22 +0000 (0:00:00.038) 0:00:38.704 ********* 2021-03-02T11:58:23.022 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.048 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.064) 0:00:38.769 ********* 2021-03-02T11:58:23.065 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.092 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.043) 0:00:38.813 ********* 2021-03-02T11:58:23.122 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.148 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.055) 0:00:38.869 ********* 2021-03-02T11:58:23.165 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.192 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.044) 0:00:38.913 ********* 2021-03-02T11:58:23.221 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.246 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.053) 0:00:38.967 ********* 2021-03-02T11:58:23.263 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-02T11:58:23.265 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.289 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.043) 0:00:39.011 ********* 2021-03-02T11:58:23.318 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.344 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.053) 0:00:39.064 ********* 2021-03-02T11:58:23.373 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-02T11:58:23.375 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.405 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.062) 0:00:39.127 ********* 2021-03-02T11:58:23.435 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.461 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.055) 0:00:39.182 ********* 2021-03-02T11:58:23.480 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-02T11:58:23.507 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.046) 0:00:39.228 ********* 2021-03-02T11:58:23.560 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.052) 0:00:39.281 ********* 2021-03-02T11:58:23.578 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-02T11:58:23.605 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.044) 0:00:39.326 ********* 2021-03-02T11:58:23.662 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.056) 0:00:39.383 ********* 2021-03-02T11:58:23.680 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-02T11:58:23.706 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.043) 0:00:39.427 ********* 2021-03-02T11:58:23.760 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.053) 0:00:39.481 ********* 2021-03-02T11:58:23.788 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-02T11:58:23.790 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.817 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.057) 0:00:39.538 ********* 2021-03-02T11:58:23.835 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.862 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.044) 0:00:39.583 ********* 2021-03-02T11:58:23.902 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.913 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.941 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.078) 0:00:39.662 ********* 2021-03-02T11:58:23.951 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.953 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-02T11:58:23.983 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:23 +0000 (0:00:00.041) 0:00:39.704 ********* 2021-03-02T11:58:24.016 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-02T11:58:24.018 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.044 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.061) 0:00:39.765 ********* 2021-03-02T11:58:24.064 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.090 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.045) 0:00:39.811 ********* 2021-03-02T11:58:24.118 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:58:24.119 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.145 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.055) 0:00:39.866 ********* 2021-03-02T11:58:24.162 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.188 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.043) 0:00:39.909 ********* 2021-03-02T11:58:24.216 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-02T11:58:24.217 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.243 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.054) 0:00:39.964 ********* 2021-03-02T11:58:24.245 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:24.262 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.290 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.047) 0:00:40.011 ********* 2021-03-02T11:58:24.325 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-02T11:58:24.327 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.363 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.072) 0:00:40.084 ********* 2021-03-02T11:58:24.391 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.418 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.055) 0:00:40.139 ********* 2021-03-02T11:58:24.448 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-02T11:58:24.449 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.476 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.058) 0:00:40.198 ********* 2021-03-02T11:58:24.495 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.528 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.051) 0:00:40.249 ********* 2021-03-02T11:58:24.570 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-02T11:58:24.572 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.595 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.067) 0:00:40.316 ********* 2021-03-02T11:58:24.613 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.637 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.041) 0:00:40.358 ********* 2021-03-02T11:58:24.677 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.706 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.069) 0:00:40.427 ********* 2021-03-02T11:58:24.724 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.750 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.043) 0:00:40.471 ********* 2021-03-02T11:58:24.778 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.804 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.053) 0:00:40.525 ********* 2021-03-02T11:58:24.840 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.848 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.043) 0:00:40.569 ********* 2021-03-02T11:58:24.876 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-02T11:58:24.877 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.902 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.053) 0:00:40.623 ********* 2021-03-02T11:58:24.933 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:24.960 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.058) 0:00:40.681 ********* 2021-03-02T11:58:24.977 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-02T11:58:24.979 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.003 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:24 +0000 (0:00:00.042) 0:00:40.724 ********* 2021-03-02T11:58:25.031 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.067 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.063) 0:00:40.788 ********* 2021-03-02T11:58:25.077 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-02T11:58:25.079 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.123 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.056) 0:00:40.844 ********* 2021-03-02T11:58:25.307 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.049) 0:00:40.894 ********* TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.056) 0:00:40.951 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.042) 0:00:40.993 ********* TASK [testnode : Stop and disable firewalld] *********************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.328 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.055) 0:00:41.049 ********* 2021-03-02T11:58:25.362 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.389 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.060) 0:00:41.110 ********* 2021-03-02T11:58:25.531 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.043) 0:00:41.154 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.055) 0:00:41.209 ********* TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.042) 0:00:41.252 ********* 2021-03-02T11:58:25.559 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.585 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.053) 0:00:41.306 ********* 2021-03-02T11:58:25.602 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-02T11:58:25.604 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.630 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.044) 0:00:41.351 ********* 2021-03-02T11:58:25.661 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.690 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.060) 0:00:41.411 ********* 2021-03-02T11:58:25.744 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-02T11:58:25.746 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.770 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.079) 0:00:41.491 ********* 2021-03-02T11:58:25.789 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.815 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.045) 0:00:41.536 ********* 2021-03-02T11:58:25.817 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:25.843 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-02T11:58:25.846 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.868 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.053) 0:00:41.589 ********* 2021-03-02T11:58:25.908 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:25.934 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.065) 0:00:41.655 ********* 2021-03-02T11:58:26.098 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:25 +0000 (0:00:00.052) 0:00:41.707 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.066) 0:00:41.773 ********* TASK [testnode : Restart abrtd] ************************************************ skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.045) 0:00:41.819 ********* 2021-03-02T11:58:26.137 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:26.163 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.065) 0:00:41.884 ********* 2021-03-02T11:58:26.307 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-02T11:58:26.376 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005183", "end": "2021-03-02 11:58:26.322088", "rc": 0, "start": "2021-03-02 11:58:26.316905", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.184) 0:00:42.069 ********* 2021-03-02T11:58:26.380 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003432", "end": "2021-03-02 11:58:26.028692", "rc": 0, "start": "2021-03-02 11:58:26.025260", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:26.407 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.059) 0:00:42.128 ********* 2021-03-02T11:58:26.436 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-02T11:58:26.471 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.063) 0:00:42.192 ********* 2021-03-02T11:58:26.565 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.093) 0:00:42.286 ********* 2021-03-02T11:58:26.942 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-02T11:58:26.944 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-02T11:58:26.967 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.402) 0:00:42.688 ********* 2021-03-02T11:58:26.975 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-02T11:58:27.002 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:26 +0000 (0:00:00.034) 0:00:42.723 ********* 2021-03-02T11:58:27.041 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-02T11:58:27.043 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:27.043 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:27.069 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:27 +0000 (0:00:00.066) 0:00:42.790 ********* 2021-03-02T11:58:27.101 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:27.145 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:27 +0000 (0:00:00.060) 0:00:42.851 ********* 2021-03-02T11:58:28.493 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-02T11:58:28.496 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-02T11:58:28.506 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-02T11:58:29.458 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-02T11:58:29.492 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:29 +0000 (0:00:02.361) 0:00:45.213 ********* 2021-03-02T11:58:29.499 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-02T11:58:29.527 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:29 +0000 (0:00:00.035) 0:00:45.249 ********* 2021-03-02T11:58:30.574 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-02T11:58:30.576 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:30.599 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:30 +0000 (0:00:01.071) 0:00:46.320 ********* 2021-03-02T11:58:30.608 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:30.641 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:30 +0000 (0:00:00.042) 0:00:46.362 ********* 2021-03-02T11:58:30.683 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-02T11:58:30.710 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:30 +0000 (0:00:00.068) 0:00:46.431 ********* 2021-03-02T11:58:30.783 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:30 +0000 (0:00:00.072) 0:00:46.504 ********* 2021-03-02T11:58:30.998 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-02T11:58:31.000 INFO:teuthology.task.ansible.out:[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036380", "end": "2021-03-02 11:58:30.652542", "rc": 0, "start": "2021-03-02 11:58:30.616162", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:31.028 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:31 +0000 (0:00:00.244) 0:00:46.749 ********* 2021-03-02T11:58:31.037 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.032449", "end": "2021-03-02 11:58:31.025966", "rc": 0, "start": "2021-03-02 11:58:30.993517", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:58:31.075 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:31 +0000 (0:00:00.047) 0:00:46.796 ********* 2021-03-02T11:58:34.144 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-02T11:58:34.146 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:34.169 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:34 +0000 (0:00:03.093) 0:00:49.890 ********* 2021-03-02T11:58:34.198 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:34.229 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:34 +0000 (0:00:00.055) 0:00:49.945 ********* 2021-03-02T11:58:43.844 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-02T11:58:43.846 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 5 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directories 2021-03-02T11:58:43.848 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 5 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2021-03-02T11:58:43.849 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-02T11:58:43.869 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:43 +0000 (0:00:09.644) 0:00:59.590 ********* 2021-03-02T11:58:44.357 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 5 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 102064 files and directories 2021-03-02T11:58:44.358 INFO:teuthology.task.ansible.out: currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 5 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Readin 2021-03-02T11:58:44.360 INFO:teuthology.task.ansible.out:g database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 102064 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-02T11:58:44.382 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:44 +0000 (0:00:00.512) 0:01:00.103 ********* 2021-03-02T11:58:44.841 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-02T11:58:44.842 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:44.866 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:44 +0000 (0:00:00.484) 0:01:00.587 ********* 2021-03-02T11:58:44.957 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-02T11:58:44.959 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:44.996 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:44 +0000 (0:00:00.130) 0:01:00.717 ********* 2021-03-02T11:58:45.364 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-02T11:58:45.366 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:45.390 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:45 +0000 (0:00:00.393) 0:01:01.111 ********* 2021-03-02T11:58:45.695 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:45 +0000 (0:00:00.099) 0:01:01.211 ********* 2021-03-02T11:58:53.840 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-02T11:58:53.842 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 5 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9333 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2021-03-02T11:58:53.844 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2021-03-02T11:58:53.846 INFO:teuthology.task.ansible.out:1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 5 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9333 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2021-03-02T11:58:53.848 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-02T11:58:53.868 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:53 +0000 (0:00:08.377) 0:01:09.589 ********* 2021-03-02T11:58:54.262 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\ngit is already the newest version (1:2.17.1-1ubuntu0.7).\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 6 newly installed, 0 to remove and 5 not upgraded.\nNeed to get 10.4 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 10.4 MB in 1s (9463 kB/s)\nSelecting previously unselected package librados2.\r\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 2021-03-02T11:58:54.266 INFO:teuthology.task.ansible.out: 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 101922 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu 2021-03-02T11:58:54.268 INFO:teuthology.task.ansible.out:1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "git is already the newest version (1:2.17.1-1ubuntu0.7).", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 6 newly installed, 0 to remove and 5 not upgraded.", "Need to get 10.4 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]", "Fetched 10.4 MB in 1s (9463 kB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Rea 2021-03-02T11:58:54.270 INFO:teuthology.task.ansible.out:ding database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 101922 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../4-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../5-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-02T11:58:54.289 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:54 +0000 (0:00:00.421) 0:01:10.010 ********* 2021-03-02T11:58:54.848 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:58:54.890 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:54 +0000 (0:00:00.601) 0:01:10.611 ********* 2021-03-02T11:58:55.139 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-02T11:58:55.141 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-02T11:58:55.168 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:55 +0000 (0:00:00.277) 0:01:10.889 ********* 2021-03-02T11:58:55.262 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-02T11:58:55.265 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:55.288 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:55 +0000 (0:00:00.120) 0:01:11.009 ********* 2021-03-02T11:58:55.299 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-02T11:58:55.300 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:58:55.331 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:55 +0000 (0:00:00.042) 0:01:11.052 ********* 2021-03-02T11:58:55.545 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-02T11:58:55.547 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-02T11:58:55.572 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:55 +0000 (0:00:00.241) 0:01:11.293 ********* 2021-03-02T11:58:55.596 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:55 +0000 (0:00:00.023) 0:01:11.317 ********* 2021-03-02T11:58:55.724 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-02T11:58:55.726 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:55.766 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:55 +0000 (0:00:00.170) 0:01:11.487 ********* 2021-03-02T11:58:56.010 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-02T11:58:56.011 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-02T11:58:56.043 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:56 +0000 (0:00:00.276) 0:01:11.763 ********* 2021-03-02T11:58:56.140 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-02T11:58:56.142 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-02T11:58:56.177 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:56 +0000 (0:00:00.134) 0:01:11.898 ********* 2021-03-02T11:58:56.442 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-02T11:58:56.445 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-02T11:58:56.467 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:56 +0000 (0:00:00.290) 0:01:12.188 ********* 2021-03-02T11:58:56.541 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-02T11:58:56.586 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:56 +0000 (0:00:00.118) 0:01:12.307 ********* 2021-03-02T11:58:56.716 INFO:teuthology.task.ansible.out: 2021-03-02T11:58:56.718 INFO:teuthology.task.ansible.out:TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:58:56.741 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:56 +0000 (0:00:00.154) 0:01:12.462 ********* 2021-03-02T11:58:56.865 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-02T11:58:56.873 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-02T11:58:56.901 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:56 +0000 (0:00:00.160) 0:01:12.622 ********* 2021-03-02T11:58:57.106 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-02T11:58:57.108 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-02 11:56:53 UTC", "ActiveEnterTimestampMonotonic": "18034071", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "nss-lookup.target basic.target system.slice network.target remote-fs.target systemd-journald.socket systemd-tmpfiles-setup.service -.mount sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-02 11:56:53 UTC", "AssertTimestampMonotonic": "17694407", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-02 11:56:53 UTC", "ConditionTimestampMonotonic": "17694407", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-02T11:58:57.110 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1434", "ExecMainStartTimestamp": "Tue 2021-03-02 11:56:53 UTC", "ExecMainStartTimestampMonotonic": "18034060", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-02 11:56:53 UTC", "InactiveExitTimestampMonotonic": "17695140", "InvocationID": "08f7dcc3ff634899bdb96bc5e49de4b8", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-02T11:58:57.112 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1434", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-02T11:58:57.186 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:56:53 UTC", "StateChangeTimestampMonotonic": "18034071", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-02 11:56:53 UTC", "WatchdogTimestampMonotonic": "18034070", "WatchdogUSec": "0"}} Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.247) 0:01:12.869 ********* TASK [testnode : Set up static IP in /etc/hosts.] ****************************** ok: [smithi036.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:58:57.202 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.053) 0:01:12.923 ********* 2021-03-02T11:58:57.232 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-02T11:58:57.235 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:57.259 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.056) 0:01:12.980 ********* 2021-03-02T11:58:57.340 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:57.365 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.106) 0:01:13.086 ********* 2021-03-02T11:58:57.457 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-02T11:58:57.619 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.115) 0:01:13.202 ********* TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.105) 0:01:13.308 ********* TASK [testnode : Stop apache2] ************************************************* changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Tue 2021-03-02 11:55:39 UTC", "ActiveEnterTimestampMonotonic": "17864569", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target system.slice nss-lookup.target basic.target systemd-tmpfiles-setup.service network.target remote-fs.target systemd-journald.socket -.mount", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-02 11:55:39 UTC", "AssertTimestampMonotonic": "17429389", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_s 2021-03-02T11:58:57.620 INFO:teuthology.task.ansible.out:ys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-02 11:55:39 UTC", "ConditionTimestampMonotonic": "17429389", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", "DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1593", "ExecMainStartTimestamp": "Tue 2021-03-02 11:55:39 UTC", "ExecMainStartTimestampMonotonic": "17864557", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "Inactive 2021-03-02T11:58:57.622 INFO:teuthology.task.ansible.out:ExitTimestamp": "Tue 2021-03-02 11:55:39 UTC", "InactiveExitTimestampMonotonic": "17429979", "InvocationID": "66f7dc22adff4f77a829b4af4d197414", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1593", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKer 2021-03-02T11:58:57.629 INFO:teuthology.task.ansible.out:nelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:55:39 UTC", "StateChangeTimestampMonotonic": "17864569", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-02 11:55:39 UTC", "WatchdogTimestampMonotonic": "17864567", "WatchdogUSec": "0"}} 2021-03-02T11:58:57.652 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.065) 0:01:13.373 ********* 2021-03-02T11:58:57.856 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-02T11:58:57.858 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:57.888 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:57 +0000 (0:00:00.235) 0:01:13.609 ********* 2021-03-02T11:58:58.041 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-02T11:58:58.043 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:58.066 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.178) 0:01:13.787 ********* 2021-03-02T11:58:58.318 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-02T11:58:58.319 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:58.376 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.310) 0:01:14.097 ********* 2021-03-02T11:58:58.639 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.044) 0:01:14.142 ********* TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.112) 0:01:14.254 ********* TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.053) 0:01:14.308 ********* TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:58.661 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.074) 0:01:14.382 ********* 2021-03-02T11:58:58.716 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.053) 0:01:14.436 ********* 2021-03-02T11:58:58.753 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:58:58.755 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:58.779 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.063) 0:01:14.500 ********* 2021-03-02T11:58:58.796 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-02T11:58:58.798 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:58.824 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.045) 0:01:14.545 ********* 2021-03-02T11:58:58.873 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-02T11:58:58.875 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:58.901 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:58 +0000 (0:00:00.076) 0:01:14.622 ********* 2021-03-02T11:58:59.088 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:59.197 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:59 +0000 (0:00:00.217) 0:01:14.839 ********* TASK [testnode : Install packages] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:59 +0000 (0:00:00.044) 0:01:14.884 ********* 2021-03-02T11:58:59.335 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-02T11:58:59.435 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:59 +0000 (0:00:00.206) 0:01:15.091 ********* TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:59.442 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:59 +0000 (0:00:00.071) 0:01:15.163 ********* 2021-03-02T11:58:59.626 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:58:59.872 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:58:59 +0000 (0:00:00.222) 0:01:15.385 ********* TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:58:59 +0000 (0:00:00.068) 0:01:15.453 ********* 2021-03-02T11:58:59.879 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:58:59.881 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.014 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.282) 0:01:15.735 ********* 2021-03-02T11:59:00.165 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.035) 0:01:15.771 ********* TASK [testnode : Remove packages] ********************************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.078) 0:01:15.849 ********* TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.036) 0:01:15.886 ********* 2021-03-02T11:59:00.196 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-02T11:59:00.197 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.222 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.056) 0:01:15.943 ********* 2021-03-02T11:59:00.239 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-02T11:59:00.241 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.267 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.045) 0:01:15.988 ********* 2021-03-02T11:59:00.299 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-02T11:59:00.301 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.339 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.071) 0:01:16.060 ********* 2021-03-02T11:59:00.362 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-02T11:59:00.382 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.431 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.089) 0:01:16.150 ********* 2021-03-02T11:59:00.447 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.474 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.045) 0:01:16.195 ********* 2021-03-02T11:59:00.546 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.573 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.098) 0:01:16.294 ********* 2021-03-02T11:59:00.575 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:00.599 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:00.601 INFO:teuthology.task.ansible.out:TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.630 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.057) 0:01:16.352 ********* 2021-03-02T11:59:00.632 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:00.714 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.739 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.108) 0:01:16.460 ********* 2021-03-02T11:59:00.740 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:00.771 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-02T11:59:00.773 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.824 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.084) 0:01:16.545 ********* 2021-03-02T11:59:00.883 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-02T11:59:00.885 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.920 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.096) 0:01:16.641 ********* 2021-03-02T11:59:00.934 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-02T11:59:00.936 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:00.963 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:00 +0000 (0:00:00.041) 0:01:16.683 ********* 2021-03-02T11:59:01.001 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-02T11:59:01.003 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.028 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.065) 0:01:16.749 ********* 2021-03-02T11:59:01.048 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-02T11:59:01.049 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.076 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.047) 0:01:16.797 ********* 2021-03-02T11:59:01.108 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-02T11:59:01.110 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.138 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.061) 0:01:16.859 ********* 2021-03-02T11:59:01.159 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-02T11:59:01.161 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.192 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.053) 0:01:16.913 ********* 2021-03-02T11:59:01.222 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.248 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.055) 0:01:16.969 ********* 2021-03-02T11:59:01.268 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.299 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.051) 0:01:17.020 ********* 2021-03-02T11:59:01.338 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-02T11:59:01.339 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.365 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.065) 0:01:17.086 ********* 2021-03-02T11:59:01.383 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-02T11:59:01.385 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.413 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.047) 0:01:17.134 ********* 2021-03-02T11:59:01.462 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-02T11:59:01.464 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.487 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.074) 0:01:17.208 ********* 2021-03-02T11:59:01.505 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-02T11:59:01.507 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.538 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.050) 0:01:17.259 ********* 2021-03-02T11:59:01.567 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.594 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.056) 0:01:17.316 ********* 2021-03-02T11:59:01.636 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-02T11:59:01.638 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.662 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.067) 0:01:17.383 ********* 2021-03-02T11:59:01.754 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-02T11:59:01.756 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.778 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.115) 0:01:17.499 ********* 2021-03-02T11:59:01.808 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-02T11:59:01.810 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-02T11:59:01.836 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.058) 0:01:17.557 ********* 2021-03-02T11:59:01.864 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-02T11:59:01.866 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:01.890 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:01 +0000 (0:00:00.054) 0:01:17.611 ********* 2021-03-02T11:59:01.997 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:02.017 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.127) 0:01:17.738 ********* 2021-03-02T11:59:02.117 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-02T11:59:02.119 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:02.142 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.124) 0:01:17.863 ********* 2021-03-02T11:59:02.365 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-02T11:59:02.388 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.246) 0:01:18.109 ********* 2021-03-02T11:59:02.405 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-02T11:59:02.434 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.045) 0:01:18.155 ********* 2021-03-02T11:59:02.548 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-02T11:59:02.550 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:02.573 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.138) 0:01:18.294 ********* 2021-03-02T11:59:02.826 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-02T11:59:02.828 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-02T11:59:02.854 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.281) 0:01:18.575 ********* 2021-03-02T11:59:02.925 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-02T11:59:02.927 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:02.949 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:02 +0000 (0:00:00.095) 0:01:18.671 ********* 2021-03-02T11:59:03.012 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-02T11:59:03.038 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:03 +0000 (0:00:00.088) 0:01:18.759 ********* 2021-03-02T11:59:03.396 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-02T11:59:03.423 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:03 +0000 (0:00:00.385) 0:01:19.144 ********* 2021-03-02T11:59:03.474 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-02T11:59:03.476 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-02T11:59:03.506 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:03 +0000 (0:00:00.083) 0:01:19.227 ********* 2021-03-02T11:59:03.859 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-02T11:59:03.861 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:03.885 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:03 +0000 (0:00:00.378) 0:01:19.606 ********* 2021-03-02T11:59:03.939 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-02T11:59:03.963 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:03 +0000 (0:00:00.078) 0:01:19.684 ********* 2021-03-02T11:59:04.179 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:59:04.212 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.247) 0:01:19.932 ********* 2021-03-02T11:59:04.328 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-02T11:59:04.330 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.352 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.140) 0:01:20.073 ********* 2021-03-02T11:59:04.360 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-02T11:59:04.362 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-02T11:59:04.394 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.042) 0:01:20.116 ********* 2021-03-02T11:59:04.478 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-02T11:59:04.480 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.516 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.120) 0:01:20.236 ********* 2021-03-02T11:59:04.587 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-02T11:59:04.589 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.619 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.103) 0:01:20.340 ********* 2021-03-02T11:59:04.644 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-02T11:59:04.646 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:59:04.672 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.053) 0:01:20.393 ********* 2021-03-02T11:59:04.721 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-02T11:59:04.723 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.726 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:04.752 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.079) 0:01:20.473 ********* 2021-03-02T11:59:04.768 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.788 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.035) 0:01:20.509 ********* 2021-03-02T11:59:04.836 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-02T11:59:04.838 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.872 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.077) 0:01:20.586 ********* 2021-03-02T11:59:04.923 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-02T11:59:04.925 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:04.949 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:04 +0000 (0:00:00.083) 0:01:20.670 ********* 2021-03-02T11:59:05.008 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:05.009 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-02T11:59:05.033 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.084) 0:01:20.754 ********* 2021-03-02T11:59:05.065 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-02T11:59:05.068 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.092 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.058) 0:01:20.813 ********* 2021-03-02T11:59:05.166 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-02T11:59:05.168 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.180 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.192 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.235 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.248 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.270 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.178) 0:01:20.991 ********* 2021-03-02T11:59:05.273 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:05.287 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:05.302 INFO:teuthology.task.ansible.out:TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-02T11:59:05.304 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi012.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.314 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.325 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.337 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.339 INFO:teuthology.task.ansible.out: skipping: [smithi012.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.361 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.362 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.373 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.413 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.142) 0:01:21.134 ********* 2021-03-02T11:59:05.437 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-02T11:59:05.439 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.479 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.066) 0:01:21.201 ********* 2021-03-02T11:59:05.624 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:05.626 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-02T11:59:05.648 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.168) 0:01:21.369 ********* 2021-03-02T11:59:05.793 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-02T11:59:05.859 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} ok: [smithi036.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} skipping: [smithi036.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.862 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.874 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.885 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.896 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.907 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:05.936 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:05 +0000 (0:00:00.287) 0:01:21.657 ********* 2021-03-02T11:59:06.042 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-02T11:59:06.044 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.043384", "end": "2021-03-02 11:59:05.976487", "rc": 0, "start": "2021-03-02 11:59:05.933103", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} 2021-03-02T11:59:06.094 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.156) 0:01:21.813 ********* 2021-03-02T11:59:06.438 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.113) 0:01:21.926 ********* TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.120) 0:01:22.046 ********* TASK [testnode : Check for teuthology-lock command] **************************** changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.005206", "end": "2021-03-02 11:59:06.260983", "rc": 0, "start": "2021-03-02 11:59:06.255777", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_3082387bba74fcd24c9700593d10418152d53c97/virtualenv/bin/teuthology-lock"]} Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.038) 0:01:22.085 ********* TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:06.441 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.077) 0:01:22.162 ********* 2021-03-02T11:59:06.490 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-02T11:59:06.762 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.076) 0:01:22.239 ********* TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.119) 0:01:22.359 ********* TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:06.774 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:06 +0000 (0:00:00.136) 0:01:22.495 ********* 2021-03-02T11:59:07.455 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:59:07.480 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:07 +0000 (0:00:00.705) 0:01:23.201 ********* 2021-03-02T11:59:07.563 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-02T11:59:07.566 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:07.588 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:07 +0000 (0:00:00.108) 0:01:23.310 ********* 2021-03-02T11:59:07.717 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-02T11:59:07.721 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item={'block_used': 13643971, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451137, 'size_available': 927543857152, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451137, "block_size": 4096, "block_total": 240095108, "block_used": 13643971, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927543857152, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-02T11:59:07.759 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:07 +0000 (0:00:00.170) 0:01:23.480 ********* 2021-03-02T11:59:07.789 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:59:08.055 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:07 +0000 (0:00:00.058) 0:01:23.538 ********* TASK [testnode : Make sure rpm dependencies are installed] ********************* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:07 +0000 (0:00:00.106) 0:01:23.645 ********* TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009898", "end": "2021-03-02 11:59:07.983650", "rc": 0, "start": "2021-03-02 11:59:07.973752", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.103) 0:01:23.748 ********* TASK [testnode : Set root disk] ************************************************ ok: [smithi036.front.sepia.ceph.com] => (item={'block_used': 13643987, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451121, 'size_available': 927543791616, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451121, "block_size": 4096, "block_total": 240095108, "block_used": 13643987, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927543791616, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-02T11:59:08.064 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.037) 0:01:23.785 ********* 2021-03-02T11:59:08.124 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-02T11:59:08.330 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'block_used': 13643971, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451137, 'size_available': 927543857152, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451137, "block_size": 4096, "block_total": 240095108, "block_used": 13643971, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927543857152, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.094) 0:01:23.880 ********* TASK [testnode : Wipe filesystems on non-root partitions] ********************** Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.121) 0:01:24.001 ********* 2021-03-02T11:59:08.330 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-02T11:59:08.331 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.009860", "end": "2021-03-02 11:59:08.004261", "rc": 0, "start": "2021-03-02 11:59:07.994401", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:08.361 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.081) 0:01:24.082 ********* 2021-03-02T11:59:09.049 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} TASK [testnode : Unmount any non-root mountpoints] ***************************** skipping: [smithi036.front.sepia.ceph.com] => (item={'block_used': 13643987, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451121, 'size_available': 927543791616, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451121, "block_size": 4096, "block_total": 240095108, "block_used": 13643987, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927543791616, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason 2021-03-02T11:59:09.051 INFO:teuthology.task.ansible.out:": "Conditional result was False"} Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.163) 0:01:24.246 ********* TASK [testnode : Wipe filesystems on non-root partitions] ********************** Tuesday 02 March 2021 11:59:08 +0000 (0:00:00.137) 0:01:24.384 ********* TASK [testnode : Zap all non-root disks] *************************************** skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:10.109 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlu1ZUyHSdpMSfME5oQ1Ds5uZDwWZ7Px3eG'], 'uuids': ['0395bf47-64e9-4fa2-95b2-792842ed33e8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.545189", "end": "2021-03-02 11:59:10.084955", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlu1ZUyHSdpMSfME5oQ1Ds5uZDwWZ7Px3eG"], "labels": [], "masters": [], "uuids": ["0395bf47-64e9-4fa2-95b2-792842ed33e8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:08.539766", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:10.111 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:10.122 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:10.478 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkQ7RSk6nNTjhlzzdpfFXtAUNsJj4uvrsG'], 'uuids': ['cd3ec109-b161-4631-83c2-b66fa562c988']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.565224", "end": "2021-03-02 11:59:10.144624", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkQ7RSk6nNTjhlzzdpfFXtAUNsJj4uvrsG"], "labels": [], "masters": [], "uuids": ["cd3ec109-b161-4631-83c2-b66fa562c988"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:08.579400", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:10.481 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:10.494 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:10.496 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:11.402 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluMJrxK0NSSx1E82Ug7UJlN2ebP2NFXeA5'], 'uuids': ['79663d3f-4e57-4fea-bc24-d8e11a057156']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.128895", "end": "2021-03-02 11:59:11.361916", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluMJrxK0NSSx1E82Ug7UJlN2ebP2NFXeA5"], "labels": [], "masters": [], "uuids": ["79663d3f-4e57-4fea-bc24-d8e11a057156"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:10.233021", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:11.426 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HP5L', 'wwn-0x5000c500871ab419'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HP5L-part1', 'wwn-0x5000c500871ab419-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ab419', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HP5L', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HP5L", "wwn-0x5000c500871ab419"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HP5L-part1", "wwn-0x5000c500871ab419-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HP5L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ab419"}}, "skip_reason": 2021-03-02T11:59:11.427 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-02T11:59:11.756 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4Nk2LsfWNRU3njnOveqhllifupB2zf9Uh1l'], 'uuids': ['39c5d510-494c-4f2a-9af7-6994d0370e62']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.124183", "end": "2021-03-02 11:59:11.419262", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4Nk2LsfWNRU3njnOveqhllifupB2zf9Uh1l"], "labels": [], "masters": [], "uuids": ["39c5d510-494c-4f2a-9af7-6994d0370e62"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:10.295079", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:11.772 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2QS', 'wwn-0x5000c500871b68bf'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2QS-part1', 'wwn-0x5000c500871b68bf-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b68bf', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2QS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2QS", "wwn-0x5000c500871b68bf"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2QS-part1", "wwn-0x5000c500871b68bf-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2QS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b68bf"}}, "skip_reason": 2021-03-02T11:59:11.773 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-02T11:59:12.729 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-YiIM2T-KlM9-KIZx-H43I-LFVl-jKjE-qR8Dcc', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331005J400BGN', 'nvme-nvme.8086-43564654353333313030354a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.139974", "end": "2021-03-02 11:59:12.708854", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-YiIM2T-KlM9-KIZx-H43I-LFVl-jKjE-qR8Dcc", "nvme-INTEL_SSDPEDMD400G4_CVFT5331005J400BGN", "nvme-nvme.8086-43564654353333313030354a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:11.568880", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-02T11:59:12.731 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:12.741 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:12.763 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:13.059 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-RQ395L-WlQY-fJcc-mfng-upcJ-R7Hw-Avou9U', 'nvme-INTEL_SSDPEDMD400G4_CVFT53300040400BGN', 'nvme-nvme.8086-43564654353333303030343034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.148917", "end": "2021-03-02 11:59:12.728729", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-RQ395L-WlQY-fJcc-mfng-upcJ-R7Hw-Avou9U", "nvme-INTEL_SSDPEDMD400G4_CVFT53300040400BGN", "nvme-nvme.8086-43564654353333303030343034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:11.579812", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-02T11:59:13.060 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:13.071 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:13.072 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:14.053 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlusQeC6w3CUtgwZnKekxQqMemuagVsGj8Q'], 'uuids': ['44b70972-e591-4988-8b50-38526e93fc90']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.156555", "end": "2021-03-02 11:59:14.039954", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlusQeC6w3CUtgwZnKekxQqMemuagVsGj8Q"], "labels": [], "masters": [], "uuids": ["44b70972-e591-4988-8b50-38526e93fc90"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:12.883399", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:14.054 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:14.066 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:14.078 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:14.332 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NknrfUfhe8St4yk1e2ngT7yDKo3IefUF38'], 'uuids': ['a896b69e-440a-4bb2-b1c2-96fe4912aeec']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.121540", "end": "2021-03-02 11:59:14.004245", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NknrfUfhe8St4yk1e2ngT7yDKo3IefUF38"], "labels": [], "masters": [], "uuids": ["a896b69e-440a-4bb2-b1c2-96fe4912aeec"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:12.882705", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:14.343 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:14.355 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:15.360 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlun2AHdhAFdPhI2B7h1n9OvWI2fn9QfJ1G'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.147860", "end": "2021-03-02 11:59:15.341771", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlun2AHdhAFdPhI2B7h1n9OvWI2fn9QfJ1G"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:14.193911", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:15.372 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:15.604 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkI42L71BwtGb6TG0ahQYvzBl8Xyvk2fhl'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.111386", "end": "2021-03-02 11:59:15.271929", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkI42L71BwtGb6TG0ahQYvzBl8Xyvk2fhl"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:14.160543", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:15.606 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:16.625 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluXUIIpqyeYvr8GvwCuUl3Chn83fog6bpe'], 'uuids': ['ac04a664-ccd5-4118-8be4-f4958d59d3c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.112822", "end": "2021-03-02 11:59:16.600761", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluXUIIpqyeYvr8GvwCuUl3Chn83fog6bpe"], "labels": [], "masters": [], "uuids": ["ac04a664-ccd5-4118-8be4-f4958d59d3c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:15.487939", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-02T11:59:16.659 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:16 +0000 (0:00:07.996) 0:01:32.380 ********* 2021-03-02T11:59:16.763 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-02T11:59:16.765 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:16.849 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-02T11:59:17.140 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkGYUPIKzhm8JXOCoNRvSgMG9Q2xLu8wOs'], 'uuids': ['f21a63b3-78bd-4f03-82d8-c6ae4df2e4b1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.107020", "end": "2021-03-02 11:59:16.522776", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkGYUPIKzhm8JXOCoNRvSgMG9Q2xLu8wOs"], "labels": [], "masters": [], "uuids": ["f21a63b3-78bd-4f03-82d8-c6ae4df2e4b1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:15.415756", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} Tuesday 02 March 2021 11:59:16 +0000 (0:00:00 2021-03-02T11:59:17.141 INFO:teuthology.task.ansible.out:.235) 0:01:32.616 ********* TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlu1ZUyHSdpMSfME5oQ1Ds5uZDwWZ7Px3eG'], 'uuids': ['0395bf47-64e9-4fa2-95b2-792842ed33e8']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.149409", "end": "2021-03-02 11:59:16.993748", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlu1ZUyHSdpMSfME5oQ1Ds5uZDwWZ7Px3eG"], "labels": [], "masters": [], "uuids": ["0395 2021-03-02T11:59:17.143 INFO:teuthology.task.ansible.out:bf47-64e9-4fa2-95b2-792842ed33e8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:16.844339", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.145125 s, 795 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.145125 s, 795 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "cha 2021-03-02T11:59:17.145 INFO:teuthology.task.ansible.out:nged": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.278 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkQ7RSk6nNTjhlzzdpfFXtAUNsJj4uvrsG'], 'uuids': ['cd3ec109-b161-4631-83c2-b66fa562c988']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.162341", "end": "2021-03-02 11:59:16.955165", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkQ7RSk6nNTjhlzzdpfFXtAUNsJj4uvrsG"], "labels": [], "masters": [], "uuids": ["cd3ec109-b161-4631-83c2-b66fa562c988"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:16.792824", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.157607 s, 732 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.157607 s, 732 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:17.326 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'maste 2021-03-02T11:59:17.328 INFO:teuthology.task.ansible.out:rs': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluMJrxK0NSSx1E82Ug7UJlN2ebP2NFXeA5'], 'uuids': ['79663d3f-4e57-4fea-bc24-d8e11a057156']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.150779", "end": "2021-03-02 11:59:17.284569", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluMJrxK0NSSx1E82Ug7UJlN2ebP2NFXeA5"], "labels": [], "masters": [], "uuids": ["79663d3f-4e57-4fea-bc24-d8e11a057156"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.133790", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14776 s, 781 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14776 s, 781 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HP5L', 'wwn-0x5000c500871ab419'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HP5L-part1', 'wwn-0x5000c500871ab419-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2 2021-03-02T11:59:17.330 INFO:teuthology.task.ansible.out:048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871ab419', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HP5L', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HP5L", "wwn-0x5000c500871ab419"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HP5L-part1", "wwn-0x5000c500871ab419-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HP5L", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871ab419"}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.486 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-YiIM2T-KlM9-KIZx-H43I-LFVl-jKjE-qR8Dcc', 'nvme-INTEL_SSDPEDMD400G4_CVFT5331005J400BGN', 'nvme-nvme.8086-43564654353333313030354a34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037235", "end": "2021-03-02 11:59:17.461114", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-YiIM2T-KlM9-KIZx-H43I-LFVl-jKjE-qR8Dcc", "nvme-INTEL_SSDPEDMD400G4_CVFT5331005J400BGN", "nvme-nvme.8086-43564654353333313030354a34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.423879", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0332038 s, 3.5 GB/s", "std 2021-03-02T11:59:17.488 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0332038 s, 3.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:17.498 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.500 INFO:teuthology.task.ansible.out: skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.502 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:17.589 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4Nk2LsfWNRU3njnOveqhllifupB2zf9Uh1l'], 'uuids': ['39c5d510-494c-4f2a-9af7-6994d0370e62']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.152260", "end": "2021-03-02 11:59:17.255194", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4Nk2LsfWNRU3njnOveqhllifupB2zf9Uh1l"], "labels": [], "masters": [], "uuids": ["39c5d510-494c-4f2a-9af7-6994d0370e62"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.102934", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.147678 s, 781 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.147678 s, 781 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:17.645 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2QS', 'wwn-0x5000c500871b68bf'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4J2QS-part1', 'wwn-0x5000c500871b68bf-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871b68bf', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4J2QS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2QS", "wwn-0x5000c500871b68bf"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4J2QS-part1", "wwn-0x5000c500871b68bf-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4J2QS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871b68bf"}}, "skip_reason": 2021-03-02T11:59:17.650 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-02T11:59:17.773 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlusQeC6w3CUtgwZnKekxQqMemuagVsGj8Q'], 'uuids': ['44b70972-e591-4988-8b50-38526e93fc90']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.149159", "end": "2021-03-02 11:59:17.753557", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlusQeC6w3CUtgwZnKekxQqMemuagVsGj8Q"], "labels": [], "masters": [], "uuids": ["44b70972-e591-4988-8b50-38526e93fc90"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.604398", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.146433 s, 788 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.146433 s, 788 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:17.781 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.791 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.792 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-RQ395L-WlQY-fJcc-mfng-upcJ-R7Hw-Avou9U', 'nvme-INTEL_SSDPEDMD400G4_CVFT53300040400BGN', 'nvme-nvme.8086-43564654353333303030343034303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.055093", "end": "2021-03-02 11:59:17.467324", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-RQ395L-WlQY-fJcc-mfng-upcJ-R7Hw-Avou9U", "nvme-INTEL_SSDPEDMD400G4_CVFT53300040400BGN", "nvme-nvme.8086-43564654353333303030343034303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.412231", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0501627 s, 2.3 GB/s", "std 2021-03-02T11:59:17.794 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0501627 s, 2.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:17.804 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:17.816 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:18.083 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NknrfUfhe8St4yk1e2ngT7yDKo3IefUF38'], 'uuids': ['a896b69e-440a-4bb2-b1c2-96fe4912aeec']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.143435", "end": "2021-03-02 11:59:17.758601", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NknrfUfhe8St4yk1e2ngT7yDKo3IefUF38"], "labels": [], "masters": [], "uuids": ["a896b69e-440a-4bb2-b1c2-96fe4912aeec"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.615166", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140485 s, 821 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140485 s, 821 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:18.259 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlun2AHdhAFdPhI2B7h1n9OvWI2fn9QfJ1G'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156664", "end": "2021-03-02 11:59:18.062929", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWlun2AHdhAFdPhI2B7h1n9OvWI2fn9QfJ1G"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.906265", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153457 s, 752 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153457 s, 752 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": 2021-03-02T11:59:18.261 INFO:teuthology.task.ansible.out:{"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rota 2021-03-02T11:59:18.263 INFO:teuthology.task.ansible.out:tional": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:18.387 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluXUIIpqyeYvr8GvwCuUl3Chn83fog6bpe'], 'uuids': ['ac04a664-ccd5-4118-8be4-f4958d59d3c5']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.163588", "end": "2021-03-02 11:59:18.364106", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-0tsNkz13qkZJUTcJ6oljUI4kuUbbcWluXUIIpqyeYvr8GvwCuUl3Chn83fog6bpe"], "labels": [], "masters": [], "uuids": ["ac04a664-ccd5-4118-8be4-f4958d59d3c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:18.200518", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.158958 s, 726 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.158958 s, 726 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:18.427 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkI42L71BwtGb6TG0ahQYvzBl8Xyvk2fhl'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.157924", "end": "2021-03-02 11:59:18.057820", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkI42L71BwtGb6TG0ahQYvzBl8Xyvk2fhl"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:17.899896", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.153172 s, 753 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.153172 s, 753 MB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": 2021-03-02T11:59:18.429 INFO:teuthology.task.ansible.out:{"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:18 +0000 (0:00:01.528) 0:01:34.145 ********* 2021-03-02T11:59:18.688 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkGYUPIKzhm8JXOCoNRvSgMG9Q2xLu8wOs'], 'uuids': ['f21a63b3-78bd-4f03-82d8-c6ae4df2e4b1']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.155688", "end": "2021-03-02 11:59:18.365103", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-LxYClNVH4d3cd5BEEEi0WghZ0H3Tk4NkGYUPIKzhm8JXOCoNRvSgMG9Q2xLu8wOs"], "labels": [], "masters": [], "uuids": ["f21a63b3-78bd-4f03-82d8-c6ae4df2e4b1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-02 11:59:18.209415", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.150896 s, 764 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.150896 s, 764 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:18.750 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:18 +0000 (0:00:00.325) 0:01:34.471 ********* 2021-03-02T11:59:18.854 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** changed: [smithi012.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.146206", "end": "2021-03-02 11:59:18.692818", "rc": 0, "start": "2021-03-02 11:59:18.546612", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Tuesday 02 March 2021 11:59:18 +0000 (0:00:00.084) 0:01:34.556 ********* 2021-03-02T11:59:19.049 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-02T11:59:19.051 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.059565", "end": "2021-03-02 11:59:19.025414", "rc": 0, "start": "2021-03-02 11:59:18.965849", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:19.074 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.239) 0:01:34.795 ********* 2021-03-02T11:59:19.102 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-02T11:59:19.186 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.142966", "end": "2021-03-02 11:59:18.720566", "rc": 0, "start": "2021-03-02 11:59:18.577600", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.080) 0:01:34.875 ********* TASK [testnode : Remove physical volumes] ************************************** 2021-03-02T11:59:19.259 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.104) 0:01:34.980 ********* 2021-03-02T11:59:19.287 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.313 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.054) 0:01:35.034 ********* 2021-03-02T11:59:19.341 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-02T11:59:19.343 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.364 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.050) 0:01:35.085 ********* 2021-03-02T11:59:19.371 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-02T11:59:19.372 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.047588", "end": "2021-03-02 11:59:19.031467", "rc": 0, "start": "2021-03-02 11:59:18.983879", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:19.398 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.033) 0:01:35.119 ********* 2021-03-02T11:59:19.405 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-02T11:59:19.407 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.431 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.033) 0:01:35.152 ********* 2021-03-02T11:59:19.439 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-02T11:59:19.469 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.037) 0:01:35.190 ********* 2021-03-02T11:59:19.475 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-02T11:59:19.477 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.501 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.032) 0:01:35.222 ********* 2021-03-02T11:59:19.507 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-02T11:59:19.509 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.546 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.044) 0:01:35.267 ********* 2021-03-02T11:59:19.577 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.606 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.059) 0:01:35.326 ********* 2021-03-02T11:59:19.633 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-02T11:59:19.634 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.662 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.056) 0:01:35.383 ********* 2021-03-02T11:59:19.669 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-02T11:59:19.671 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-02T11:59:19.696 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.033) 0:01:35.417 ********* 2021-03-02T11:59:19.704 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-02T11:59:19.706 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.731 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.035) 0:01:35.452 ********* 2021-03-02T11:59:19.737 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-02T11:59:19.739 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'block_used': 13643971, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451137, 'size_available': 927543857152, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451137, "block_size": 4096, "block_total": 240095108, "block_used": 13643971, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927543857152, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.765 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.033) 0:01:35.486 ********* 2021-03-02T11:59:19.793 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-02T11:59:19.795 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.816 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.050) 0:01:35.537 ********* 2021-03-02T11:59:19.845 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:19.946 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.055) 0:01:35.592 ********* TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi036.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.033) 0:01:35.626 ********* TASK [testnode : set_fact] ***************************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.032) 0:01:35.658 ********* 2021-03-02T11:59:19.955 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi036.front.sepia.ceph.com] => (item={'block_used': 13643987, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226451121, 'size_available': 927543791616, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 113943, 'block_size': 4096, 'inode_available': 60941033}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226451121, "block_size": 4096, "block_total": 240095108, "block_used": 13643987, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60941033, "inode_total": 61054976, "inode_used": 113943, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927543791616, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:19.980 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:19 +0000 (0:00:00.043) 0:01:35.701 ********* 2021-03-02T11:59:19.988 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-02T11:59:19.989 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.016 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.035) 0:01:35.737 ********* 2021-03-02T11:59:20.023 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-02T11:59:20.026 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.052 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.036) 0:01:35.773 ********* 2021-03-02T11:59:20.059 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-02T11:59:20.060 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.087 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.033) 0:01:35.807 ********* 2021-03-02T11:59:20.093 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:20.095 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.122 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.035) 0:01:35.843 ********* 2021-03-02T11:59:20.128 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-02T11:59:20.130 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.156 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.033) 0:01:35.876 ********* 2021-03-02T11:59:20.164 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:20.264 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.038) 0:01:35.915 ********* TASK [testnode : Create volume_groups dict] ************************************ skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.069) 0:01:35.985 ********* 2021-03-02T11:59:20.305 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-02T11:59:20.307 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.331 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.067) 0:01:36.052 ********* 2021-03-02T11:59:20.371 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:20.395 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.063) 0:01:36.116 ********* 2021-03-02T11:59:20.779 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-02T11:59:20.803 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.407) 0:01:36.524 ********* 2021-03-02T11:59:20.811 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-02T11:59:20.837 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:20 +0000 (0:00:00.034) 0:01:36.559 ********* 2021-03-02T11:59:21.384 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-02T11:59:21.386 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:21.417 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:21.635 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:21.932 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:21.944 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:22.090 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:22.275 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:22.346 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:22.374 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:22 +0000 (0:00:01.536) 0:01:38.095 ********* 2021-03-02T11:59:22.552 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-02T11:59:22.576 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:22 +0000 (0:00:00.201) 0:01:38.297 ********* 2021-03-02T11:59:22.593 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-02T11:59:22.595 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-02T11:59:22.626 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:22 +0000 (0:00:00.050) 0:01:38.347 ********* 2021-03-02T11:59:22.750 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:22.804 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-02T11:59:22.806 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-02T11:59:22.829 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:22 +0000 (0:00:00.202) 0:01:38.550 ********* 2021-03-02T11:59:22.900 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-02T11:59:22.902 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:22.998 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:23.043 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:23.141 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:23.184 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:23.206 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:23.233 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.404) 0:01:38.954 ********* 2021-03-02T11:59:23.265 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:23.311 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.077) 0:01:39.031 ********* 2021-03-02T11:59:23.321 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-02T11:59:23.323 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:23.345 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-02T11:59:23.346 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:23.380 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.069) 0:01:39.101 ********* 2021-03-02T11:59:23.398 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-02T11:59:23.400 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-02T11:59:23.411 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:23.412 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:23.450 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.070) 0:01:39.171 ********* 2021-03-02T11:59:23.494 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-02T11:59:23.497 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:23.523 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.072) 0:01:39.244 ********* 2021-03-02T11:59:23.551 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-02T11:59:23.553 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:23.576 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.052) 0:01:39.297 ********* 2021-03-02T11:59:23.584 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:23.620 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.043) 0:01:39.341 ********* 2021-03-02T11:59:23.671 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-02T11:59:23.673 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-02T11:59:23.698 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.078) 0:01:39.419 ********* 2021-03-02T11:59:23.739 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-02T11:59:23.741 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:23.765 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.066) 0:01:39.486 ********* 2021-03-02T11:59:23.803 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-02T11:59:23.805 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-02T11:59:23.828 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:23 +0000 (0:00:00.063) 0:01:39.549 ********* 2021-03-02T11:59:24.203 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-02T11:59:24.205 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true} changed: [smithi012.front.sepia.ceph.com] => {"changed": true} 2021-03-02T11:59:24.229 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.401) 0:01:39.950 ********* 2021-03-02T11:59:24.254 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.024) 0:01:39.975 ********* 2021-03-02T11:59:24.715 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-02T11:59:24.717 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-02T11:59:24.742 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.487) 0:01:40.463 ********* 2021-03-02T11:59:24.764 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.022) 0:01:40.485 ********* 2021-03-02T11:59:24.770 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-02T11:59:24.775 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:24.805 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.040) 0:01:40.526 ********* 2021-03-02T11:59:24.813 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:24.837 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.031) 0:01:40.558 ********* 2021-03-02T11:59:24.843 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-02T11:59:24.845 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:24.871 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.033) 0:01:40.592 ********* 2021-03-02T11:59:24.877 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:24.904 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.033) 0:01:40.625 ********* 2021-03-02T11:59:24.910 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-02T11:59:24.912 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:24.936 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.032) 0:01:40.657 ********* 2021-03-02T11:59:24.947 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:24.974 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:24 +0000 (0:00:00.037) 0:01:40.695 ********* 2021-03-02T11:59:24.982 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-02T11:59:24.983 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:25.007 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.032) 0:01:40.728 ********* 2021-03-02T11:59:25.015 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:25.040 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.033) 0:01:40.761 ********* 2021-03-02T11:59:25.174 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-02T11:59:25.199 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.158) 0:01:40.920 ********* 2021-03-02T11:59:25.201 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:25.208 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-02T11:59:25.236 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.036) 0:01:40.957 ********* 2021-03-02T11:59:25.361 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-02T11:59:25.362 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:25.386 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.149) 0:01:41.107 ********* 2021-03-02T11:59:25.394 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:25.421 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.034) 0:01:41.142 ********* 2021-03-02T11:59:25.745 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-02T11:59:25.747 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-02T11:59:25.769 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.348) 0:01:41.490 ********* 2021-03-02T11:59:25.788 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-02T11:59:25.816 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:25 +0000 (0:00:00.046) 0:01:41.537 ********* 2021-03-02T11:59:26.121 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-02T11:59:26.123 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-02T11:59:26.153 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:26 +0000 (0:00:00.337) 0:01:41.874 ********* 2021-03-02T11:59:26.183 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-02T11:59:26.210 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:26 +0000 (0:00:00.056) 0:01:41.931 ********* 2021-03-02T11:59:27.104 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-02T11:59:27.106 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:59:27.129 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.919) 0:01:42.851 ********* 2021-03-02T11:59:27.147 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:59:27.174 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.043) 0:01:42.895 ********* 2021-03-02T11:59:27.321 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-02T11:59:27.608 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.047034", "end": "2021-03-02 11:59:27.303982", "rc": 0, "start": "2021-03-02 11:59:27.256948", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.173) 0:01:43.068 ********* ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.044323", "end": "2021-03-02 11:59:27.327576", "rc": 0, "start": "2021-03-02 11:59:27.283253", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.033) 0:01:43.102 ********* TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.037) 0:01:43.139 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.035) 0:01:43.174 ********* TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.032) 0:01:43.207 ********* skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:27 +0000 (0:00:00.032) 0:01:43.240 ********* 2021-03-02T11:59:28.378 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-02T11:59:28.379 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:59:28.402 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.883) 0:01:44.123 ********* 2021-03-02T11:59:28.430 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-02T11:59:28.432 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Install ntp package on apt based systems.] ******************** ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:59:28.454 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.051) 0:01:44.175 ********* 2021-03-02T11:59:28.476 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.021) 0:01:44.197 ********* 2021-03-02T11:59:28.503 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-02T11:59:28.505 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:28.528 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.052) 0:01:44.249 ********* 2021-03-02T11:59:28.828 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-02T11:59:28.830 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-02T11:59:28.853 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.324) 0:01:44.574 ********* 2021-03-02T11:59:28.881 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-02T11:59:28.883 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:28.910 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.056) 0:01:44.631 ********* 2021-03-02T11:59:28.933 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.023) 0:01:44.654 ********* 2021-03-02T11:59:28.962 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:28.985 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:28 +0000 (0:00:00.052) 0:01:44.706 ********* 2021-03-02T11:59:29.157 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-02T11:59:29.160 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-02 11:56:59 UTC", "ActiveEnterTimestampMonotonic": "23352618", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice systemd-journald.socket -.mount systemd-tmpfiles-setup.service sysinit.target basic.target network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-02 11:56:52 UTC", "AssertTimestampMonotonic": "17080077", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-02 11:56:52 UTC", "ConditionTimestampMonotonic": "17080077", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-02T11:59:29.161 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2011", "ExecMainStartTimestamp": "Tue 2021-03-02 11:56:59 UTC", "ExecMainStartTimestampMonotonic": "23352595", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-02 11:56:52 UTC", "InactiveExitTimestampMonotonic": "17080829", "InvocationID": "37baacccabf44e528091f0d94dc43e8e", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-02T11:59:29.163 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2011", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:56:59 UTC", "S 2021-03-02T11:59:29.165 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23352618", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-02 11:56:59 UTC", "WatchdogTimestampMonotonic": "23352616", "WatchdogUSec": "0"}} 2021-03-02T11:59:29.189 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.203) 0:01:44.910 ********* 2021-03-02T11:59:29.220 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:29.222 INFO:teuthology.task.ansible.out:TASK [testnode : set_fact] ***************************************************** ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-02T11:59:29.254 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.064) 0:01:44.974 ********* 2021-03-02T11:59:29.256 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:29.264 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-02T11:59:29.266 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Tue 2021-03-02 11:55:44 UTC", "ActiveEnterTimestampMonotonic": "23216536", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target system.slice sysinit.target basic.target systemd-tmpfiles-setup.service -.mount systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Tue 2021-03-02 11:55:38 UTC", "AssertTimestampMonotonic": "16812115", "Before": "multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Tue 2021-03-02 11:55:38 UTC", "ConditionTimestampMonotonic": "16812114", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-02T11:59:29.268 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "2167", "ExecMainStartTimestamp": "Tue 2021-03-02 11:55:44 UTC", "ExecMainStartTimestampMonotonic": "23216483", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Tue 2021-03-02 11:55:38 UTC", "InactiveExitTimestampMonotonic": "16812847", "InvocationID": "7a97ed3cd3ad431cb7b15b9a4a37ed29", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-02T11:59:29.270 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "2167", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:55:44 UTC", "S 2021-03-02T11:59:29.271 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23216536", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Tue 2021-03-02 11:55:44 UTC", "WatchdogTimestampMonotonic": "23216533", "WatchdogUSec": "0"}} 2021-03-02T11:59:29.293 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.039) 0:01:45.014 ********* 2021-03-02T11:59:29.300 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:29.302 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:29.337 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.043) 0:01:45.058 ********* 2021-03-02T11:59:29.423 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.043) 0:01:45.101 ********* TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.037) 0:01:45.138 ********* 2021-03-02T11:59:29.424 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:29.424 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:29.450 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.032) 0:01:45.171 ********* 2021-03-02T11:59:29.478 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-02T11:59:29.480 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:29.502 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:29 +0000 (0:00:00.051) 0:01:45.222 ********* 2021-03-02T11:59:30.366 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-02T11:59:30.368 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:59:30.403 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:30 +0000 (0:00:00.901) 0:01:46.124 ********* 2021-03-02T11:59:30.434 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-02T11:59:30.436 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:30.458 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:30 +0000 (0:00:00.055) 0:01:46.179 ********* 2021-03-02T11:59:30.465 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-02T11:59:30.467 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:59:30.498 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:30 +0000 (0:00:00.040) 0:01:46.219 ********* 2021-03-02T11:59:30.528 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-02T11:59:30.530 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:30.552 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:30 +0000 (0:00:00.053) 0:01:46.273 ********* 2021-03-02T11:59:30.654 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-02T11:59:30.656 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-02T11:59:30.697 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:30 +0000 (0:00:00.145) 0:01:46.418 ********* 2021-03-02T11:59:30.711 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-02T11:59:30.745 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:30 +0000 (0:00:00.048) 0:01:46.466 ********* 2021-03-02T11:59:31.058 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-02T11:59:31.060 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-02T11:59:31.083 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:31 +0000 (0:00:00.337) 0:01:46.804 ********* 2021-03-02T11:59:31.111 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-02T11:59:31.135 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:31 +0000 (0:00:00.052) 0:01:46.856 ********* 2021-03-02T11:59:32.137 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-02T11:59:32.140 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-02T11:59:32.163 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:32 +0000 (0:00:01.027) 0:01:47.884 ********* 2021-03-02T11:59:32.192 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-02T11:59:32.218 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:32 +0000 (0:00:00.055) 0:01:47.939 ********* 2021-03-02T11:59:32.707 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi036.front.sepia.ceph.com] => {"changed": false} 2021-03-02T11:59:32.731 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:32 +0000 (0:00:00.512) 0:01:48.452 ********* 2021-03-02T11:59:32.769 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:32.771 INFO:teuthology.task.ansible.out:TASK [testnode : Set interface] ************************************************ ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-02T11:59:32.795 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:32 +0000 (0:00:00.063) 0:01:48.516 ********* 2021-03-02T11:59:32.825 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-02T11:59:32.827 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false} 2021-03-02T11:59:32.855 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:32 +0000 (0:00:00.060) 0:01:48.576 ********* 2021-03-02T11:59:32.894 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-02T11:59:32.896 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-02T11:59:32.919 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:32 +0000 (0:00:00.063) 0:01:48.640 ********* 2021-03-02T11:59:32.991 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-02T11:59:32.993 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1614686154.236287, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1614686154.232285, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1614686154.232285, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-02T11:59:33.019 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.100) 0:01:48.740 ********* 2021-03-02T11:59:33.095 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1614686228.4888573, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1614686228.4848592, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1614686228.4848592, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-02T11:59:33.124 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.105) 0:01:48.845 ********* 2021-03-02T11:59:33.494 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-02T11:59:33.496 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "msg": ""} ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-02T11:59:33.522 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.397) 0:01:49.243 ********* 2021-03-02T11:59:33.525 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:33.556 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.033) 0:01:49.277 ********* 2021-03-02T11:59:33.558 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:33.579 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-02T11:59:33.581 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-02T11:59:33.604 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.048) 0:01:49.326 ********* 2021-03-02T11:59:33.616 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-02T11:59:33.656 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.051) 0:01:49.377 ********* 2021-03-02T11:59:33.664 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-02T11:59:33.666 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:33.693 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.036) 0:01:49.414 ********* 2021-03-02T11:59:33.710 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:33.735 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.042) 0:01:49.457 ********* 2021-03-02T11:59:33.853 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-02T11:59:33.855 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-02T11:59:33.887 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.151) 0:01:49.608 ********* 2021-03-02T11:59:33.905 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-02T11:59:33.934 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:33 +0000 (0:00:00.046) 0:01:49.655 ********* 2021-03-02T11:59:34.049 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-02T11:59:34.051 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-02T11:59:34.077 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.143) 0:01:49.798 ********* 2021-03-02T11:59:34.086 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-02T11:59:34.111 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.033) 0:01:49.832 ********* 2021-03-02T11:59:34.119 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-02T11:59:34.121 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.151 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.040) 0:01:49.872 ********* 2021-03-02T11:59:34.159 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.188 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.036) 0:01:49.909 ********* 2021-03-02T11:59:34.202 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-02T11:59:34.204 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.204 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.205 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.205 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.238 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.050) 0:01:49.959 ********* 2021-03-02T11:59:34.249 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.250 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.251 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.267 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.299 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.061) 0:01:50.020 ********* 2021-03-02T11:59:34.308 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-02T11:59:34.309 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.310 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.311 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.341 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.041) 0:01:50.062 ********* 2021-03-02T11:59:34.352 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.364 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.366 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi036.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.366 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.367 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi036.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.399 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.057) 0:01:50.120 ********* 2021-03-02T11:59:34.429 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.030) 0:01:50.150 ********* 2021-03-02T11:59:34.439 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-02T11:59:34.441 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.473 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.044) 0:01:50.195 ********* 2021-03-02T11:59:34.482 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-02T11:59:34.485 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.513 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.039) 0:01:50.234 ********* 2021-03-02T11:59:34.521 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.562 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.049) 0:01:50.283 ********* 2021-03-02T11:59:34.573 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-02T11:59:34.575 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.609 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.047) 0:01:50.331 ********* 2021-03-02T11:59:34.616 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-02T11:59:34.618 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.642 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.032) 0:01:50.364 ********* 2021-03-02T11:59:34.650 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-02T11:59:34.651 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.678 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.035) 0:01:50.399 ********* 2021-03-02T11:59:34.685 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-02T11:59:34.687 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.714 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.036) 0:01:50.435 ********* 2021-03-02T11:59:34.722 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-02T11:59:34.724 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.753 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.038) 0:01:50.474 ********* 2021-03-02T11:59:34.761 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-02T11:59:34.763 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.789 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.036) 0:01:50.510 ********* 2021-03-02T11:59:34.796 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-02T11:59:34.798 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.826 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.037) 0:01:50.547 ********* 2021-03-02T11:59:34.833 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-02T11:59:34.840 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.871 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.044) 0:01:50.592 ********* 2021-03-02T11:59:34.878 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-02T11:59:34.880 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.914 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.043) 0:01:50.635 ********* 2021-03-02T11:59:34.924 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.952 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.037) 0:01:50.673 ********* 2021-03-02T11:59:34.970 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:34.999 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:34 +0000 (0:00:00.047) 0:01:50.720 ********* 2021-03-02T11:59:35.028 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-02T11:59:35.030 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:35.052 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.053) 0:01:50.773 ********* 2021-03-02T11:59:35.136 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi012.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011582", "end": "2021-03-02 11:59:35.095303", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-02 11:59:35.083721", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.218 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.011171", "end": "2021-03-02 11:59:35.187049", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-02 11:59:35.175878", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.261 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006024", "end": "2021-03-02 11:59:35.227568", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-02 11:59:35.221544", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.346 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006128", "end": "2021-03-02 11:59:35.317872", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-02 11:59:35.311744", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.378 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.003633", "end": "2021-03-02 11:59:35.353083", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-02 11:59:35.349450", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.474 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005606", "end": "2021-03-02 11:59:35.448217", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-02 11:59:35.442611", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.537 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.019637", "end": "2021-03-02 11:59:35.506998", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-02 11:59:35.487361", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.566 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.513) 0:01:51.287 ********* 2021-03-02T11:59:35.593 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-02T11:59:35.620 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.054) 0:01:51.341 ********* 2021-03-02T11:59:35.628 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-02T11:59:35.630 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.023833", "end": "2021-03-02 11:59:35.589272", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-02 11:59:35.565439", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:35.670 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.049) 0:01:51.391 ********* 2021-03-02T11:59:35.712 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-02T11:59:35.737 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.066) 0:01:51.458 ********* 2021-03-02T11:59:35.798 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-02T11:59:35.800 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-02T11:59:35.880 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-02T11:59:35.951 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-02T11:59:35.970 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.233) 0:01:51.691 ********* 2021-03-02T11:59:35.996 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:35 +0000 (0:00:00.025) 0:01:51.717 ********* 2021-03-02T11:59:36.693 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-02T11:59:36.695 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] 2021-03-02T11:59:36.749 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:36 +0000 (0:00:00.753) 0:01:52.470 ********* 2021-03-02T11:59:36.790 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:36.792 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-02T11:59:36.815 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:36 +0000 (0:00:00.066) 0:01:52.536 ********* 2021-03-02T11:59:36.845 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:36.847 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:36.885 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:36 +0000 (0:00:00.069) 0:01:52.605 ********* 2021-03-02T11:59:36.939 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-02T11:59:36.941 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-02T11:59:36.966 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:36 +0000 (0:00:00.081) 0:01:52.687 ********* 2021-03-02T11:59:37.772 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-02T11:59:37.772 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] 2021-03-02T11:59:37.774 INFO:teuthology.task.ansible.out: 2021-03-02T11:59:37.844 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:37 +0000 (0:00:00.878) 0:01:53.566 ********* 2021-03-02T11:59:37.882 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:37.884 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-02T11:59:37.907 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:37 +0000 (0:00:00.061) 0:01:53.628 ********* 2021-03-02T11:59:37.934 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:37.939 INFO:teuthology.task.ansible.out:skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:37.958 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:37 +0000 (0:00:00.051) 0:01:53.679 ********* 2021-03-02T11:59:37.985 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-02T11:59:37.986 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"cache_update_time": 1614686248, "cache_updated": false, "changed": false} 2021-03-02T11:59:38.011 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:38 +0000 (0:00:00.052) 0:01:53.732 ********* 2021-03-02T11:59:38.019 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-02T11:59:38.020 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-02T11:59:38.048 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:38 +0000 (0:00:00.037) 0:01:53.769 ********* 2021-03-02T11:59:38.055 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:38.057 INFO:teuthology.task.ansible.out:skipping: [smithi012.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:38.092 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:38 +0000 (0:00:00.043) 0:01:53.813 ********* 2021-03-02T11:59:38.133 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:38.135 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-02T11:59:38.158 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:38 +0000 (0:00:00.065) 0:01:53.879 ********* 2021-03-02T11:59:38.314 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-02T11:59:38.316 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:38.444 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:38.809 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:38 +0000 (0:00:00.317) 0:01:54.196 ********* 2021-03-02T11:59:38.901 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-02T11:59:38.903 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686378.539769-10580-122541527541299/source", "state": "file", "uid": 0} 2021-03-02T11:59:38.998 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-02T11:59:38.999 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"cache_update_time": 1614686253, "cache_updated": false, "changed": false} 2021-03-02T11:59:39.023 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:39 +0000 (0:00:00.547) 0:01:54.744 ********* 2021-03-02T11:59:39.054 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi036.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-02T11:59:39.081 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:39 +0000 (0:00:00.058) 0:01:54.802 ********* 2021-03-02T11:59:39.127 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-02T11:59:39.544 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} Tuesday 02 March 2021 11:59:39 +0000 (0:00:00.227) 0:01:55.029 ********* TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** changed: [smithi012.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686378.934044-10580-160642098537840/source", "state": "file", "uid": 0} Tuesday 02 March 2021 11:59:39 +0000 (0:00:00.039) 0:01:55.069 ********* TASK [container-host : Create container_mirror_cert_paths] ********************* changed: [smithi036.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:39.643 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-02T11:59:39.669 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:39 +0000 (0:00:00.320) 0:01:55.390 ********* 2021-03-02T11:59:40.175 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-02T11:59:40.177 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686379.7470303-10777-59366852202261/source", "state": "file", "uid": 0} 2021-03-02T11:59:40.580 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686380.1945064-10777-98446937870960/source", "state": "file", "uid": 0} 2021-03-02T11:59:40.785 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:40 +0000 (0:00:00.936) 0:01:56.327 ********* 2021-03-02T11:59:43.049 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-02T11:59:43.051 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-WDvNAp-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-WDvNAp-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-02T11:59:43.052 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-02T11:59:43.080 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:43 +0000 (0:00:02.474) 0:01:58.801 ********* 2021-03-02T11:59:43.311 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-02T11:59:43.313 INFO:teuthology.task.ansible.out:ok: [smithi012.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-02T11:59:43.342 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:43 +0000 (0:00:00.262) 0:01:59.063 ********* 2021-03-02T11:59:44.099 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-02T11:59:44.101 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686383.462789-11429-4069607064450/source", "state": "file", "uid": 0} 2021-03-02T11:59:44.124 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:44 +0000 (0:00:00.782) 0:01:59.845 ********* 2021-03-02T11:59:44.158 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-02T11:59:44.159 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-exgLN1-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-exgLN1-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-02T11:59:44.161 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-02T11:59:44.185 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:44 +0000 (0:00:00.060) 0:01:59.906 ********* 2021-03-02T11:59:44.351 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-02T11:59:44.353 INFO:teuthology.task.ansible.out:ok: [smithi036.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-02T11:59:44.381 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:44 +0000 (0:00:00.196) 0:02:00.102 ********* 2021-03-02T11:59:44.420 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-02T11:59:44.422 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.124024", "end": "2021-03-02 11:59:44.390200", "rc": 0, "start": "2021-03-02 11:59:44.266176", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:44.453 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:44 +0000 (0:00:00.071) 0:02:00.174 ********* 2021-03-02T11:59:44.848 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-02T11:59:44.850 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1614686384.4390116-11594-180436515964446/source", "state": "file", "uid": 0} 2021-03-02T11:59:44.874 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:44 +0000 (0:00:00.421) 0:02:00.595 ********* 2021-03-02T11:59:45.196 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-02T11:59:45.198 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.125055", "end": "2021-03-02 11:59:45.168875", "rc": 0, "start": "2021-03-02 11:59:45.043820", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-02T11:59:45.225 INFO:teuthology.task.ansible.out:Tuesday 02 March 2021 11:59:45 +0000 (0:00:00.351) 0:02:00.946 ********* 2021-03-02T11:59:47.841 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-02T11:59:47.843 INFO:teuthology.task.ansible.out:changed: [smithi012.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "firewalld.service network-online.target systemd-journald.socket basic.target system.slice containerd.service sysinit.target docker.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-02T11:59:47.845 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-03-02T11:59:47.847 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "docker.socket system.slice sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:57:15 UTC", "StateChangeTimestampMonotonic": "40028231", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-02T11:59:47.848 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-02T11:59:48.512 INFO:teuthology.task.ansible.out:changed: [smithi036.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice docker.socket network-online.target systemd-journald.socket containerd.service basic.target firewalld.service sysinit.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-02T11:59:48.513 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "12805 2021-03-02T11:59:48.515 INFO:teuthology.task.ansible.out:3", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target docker.socket", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Tue 2021-03-02 11:56:00 UTC", "StateChangeTimestampMonotonic": "39248285", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "Su 2021-03-02T11:59:48.517 INFO:teuthology.task.ansible.out:ccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-02T11:59:48.594 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-02T11:59:48.596 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-02T11:59:48.597 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi012.front.sepia.ceph.com : ok=120 changed=32 unreachable=0 failed=0 skipped=190 rescued=0 ignored=0 smithi036.front.sepia.ceph.com : ok=120 changed=32 unreachable=0 failed=0 skipped=190 rescued=0 ignored=0 Tuesday 02 March 2021 11:59:48 +0000 (0:00:03.371) 0:02:04.318 ********* =============================================================================== testnode : Ensure packages are not present. ----------------------------- 9.64s testnode : Install packages --------------------------------------------- 8.38s testnode : Zap all non-root disks --------------------------------------- 8.00s container-host : Restart docker service --------------------------------- 3.37s common : Update apt cache ----------------------------------------------- 3.15s testnode : Update apt cache. -------------------------------------------- 3.09s container-host : Install registries-conf-ctl ---------------------------- 2.47s 2021-03-02T11:59:48.598 INFO:teuthology.task.ansible.out:testnode : Install apt keys --------------------------------------------- 2.36s Gathering Facts --------------------------------------------------------- 1.70s testnode : Create logical volume(s) ------------------------------------- 1.54s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.53s testnode : Install python-apt ------------------------------------------- 1.07s testnode : Install packages via pip ------------------------------------- 1.03s common : Install Kerberos Packages (Debian) ----------------------------- 1.02s common : Make sure smartmontools is installed --------------------------- 0.95s common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/. --- 0.95s container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert --- 0 2021-03-02T11:59:48.600 INFO:teuthology.task.ansible.out:.94s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.94s testnode : Ensure perl-doc is installed on apt systems. ----------------- 0.92s testnode : Install python-pip on apt based systems. --------------------- 0.90s 2021-03-02T11:59:48.984 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi012.front.sepia.ceph.com'), Remote(name='ubuntu@smithi036.front.sepia.ceph.com')] 2021-03-02T11:59:48.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-02T11:59:48.988 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi012.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-02T11:59:49.110 DEBUG:teuthology.orchestra.run.smithi012:> true 2021-03-02T11:59:49.319 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-02T11:59:49.389 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi036.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-02T11:59:49.510 DEBUG:teuthology.orchestra.run.smithi036:> true 2021-03-02T11:59:49.719 INFO:teuthology.run_tasks:Running task clock... 2021-03-02T11:59:49.730 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-02T11:59:49.731 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-02T11:59:49.732 DEBUG:teuthology.orchestra.run.smithi012:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-02T11:59:49.786 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-02T11:59:49.787 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Command line: ntpd -gq 2021-03-02T11:59:49.788 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: proto: precision = 0.100 usec (-23) 2021-03-02T11:59:49.788 INFO:teuthology.orchestra.run.smithi012.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-02T11:59:49.789 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-02T11:59:49.789 INFO:teuthology.orchestra.run.smithi012.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-02T11:59:49.790 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: restrict ::: KOD does nothing without LIMITED. 2021-03-02T11:59:49.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listen and drop on 0 v6wildcard [::]:123 2021-03-02T11:59:49.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-02T11:59:49.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listen normally on 2 lo 127.0.0.1:123 2021-03-02T11:59:49.791 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listen normally on 3 ens1f0 172.21.15.12:123 2021-03-02T11:59:49.792 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listen normally on 4 lo [::1]:123 2021-03-02T11:59:49.792 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:25ea%3]:123 2021-03-02T11:59:49.792 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:49 ntpd[7369]: Listening on routing socket on fd #22 for interface updates 2021-03-02T11:59:50.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:50 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:50.788 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:50 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:50.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:50 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:50.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:50 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:51.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:51.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:51.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:51.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:51.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:51.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:51.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:51.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:51 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:52.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:52 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:52.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:52 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:52.789 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:52 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:52.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:52 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:53.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:53.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:53.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:53.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:53.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:53.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:53.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:53.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:53.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:53.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:53.795 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:53.796 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:53 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:54.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:54 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:54.790 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:54 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:54.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:54 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:54.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:54 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:55.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:55.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:55.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:55.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:55.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:55.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:55.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:55.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:55.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:55.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:55.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:55.794 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:55 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:56.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:56 ntpd[7369]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:56.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:56 ntpd[7369]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:56.791 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:56 ntpd[7369]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:56.792 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:56 ntpd[7369]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:56.792 INFO:teuthology.orchestra.run.smithi012.stdout: 2 Mar 11:59:56 ntpd[7369]: ntpd: time slew +0.013513 s 2021-03-02T11:59:56.792 INFO:teuthology.orchestra.run.smithi012.stdout:ntpd: time slew +0.013513s 2021-03-02T11:59:56.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:56 ntpd[7369]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-02T11:59:56.793 INFO:teuthology.orchestra.run.smithi012.stderr: 2 Mar 11:59:56 ntpd[7369]: can't open /var/log/ntpstats/loopstats.20210302: Permission denied 2021-03-02T11:59:56.844 INFO:teuthology.orchestra.run.smithi012.stdout: remote refid st t when poll reach delay offset jitter 2021-03-02T11:59:56.844 INFO:teuthology.orchestra.run.smithi012.stdout:============================================================================== 2021-03-02T11:59:56.844 INFO:teuthology.orchestra.run.smithi012.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T11:59:56.844 INFO:teuthology.orchestra.run.smithi012.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T11:59:56.845 INFO:teuthology.orchestra.run.smithi012.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T11:59:56.845 INFO:teuthology.orchestra.run.smithi012.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T11:59:56.846 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-02T11:59:56.846 DEBUG:teuthology.orchestra.run.smithi036:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-02T11:59:56.911 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-02T11:59:56.912 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Command line: ntpd -gq 2021-03-02T11:59:56.913 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: proto: precision = 0.137 usec (-23) 2021-03-02T11:59:56.913 INFO:teuthology.orchestra.run.smithi036.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-02T11:59:56.914 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-02T11:59:56.914 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: restrict ::: KOD does nothing without LIMITED. 2021-03-02T11:59:56.915 INFO:teuthology.orchestra.run.smithi036.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-02T11:59:56.916 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listen and drop on 0 v6wildcard [::]:123 2021-03-02T11:59:56.916 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-02T11:59:56.918 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listen normally on 2 lo 127.0.0.1:123 2021-03-02T11:59:56.918 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listen normally on 3 ens1f0 172.21.15.36:123 2021-03-02T11:59:56.918 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listen normally on 4 lo [::1]:123 2021-03-02T11:59:56.918 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listen normally on 5 ens1f0 [fe80::ae1f:6bff:fef8:2542%3]:123 2021-03-02T11:59:56.919 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 11:59:56 ntpd[7524]: Listening on routing socket on fd #22 for interface updates 2021-03-02T11:59:57.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:57.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:57.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:57.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:57.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:57.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:57.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:57.915 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:57 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:59.912 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:59.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:59.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:59.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T11:59:59.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T11:59:59.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T11:59:59.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T11:59:59.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 11:59:59 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:00.912 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:00 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T12:00:00.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:00 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T12:00:00.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:00 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T12:00:00.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:00 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:01.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T12:00:01.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T12:00:01.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T12:00:01.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:01.915 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T12:00:01.915 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T12:00:01.915 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T12:00:01.915 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:01.916 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T12:00:01.916 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T12:00:01.916 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T12:00:01.916 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:01 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:02.912 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:02 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T12:00:02.912 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:02 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T12:00:02.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:02 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T12:00:02.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:02 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:03.912 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:03 ntpd[7524]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-02T12:00:03.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:03 ntpd[7524]: can't open /var/log/ntpstats/rawstats.20210302: Permission denied 2021-03-02T12:00:03.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:03 ntpd[7524]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-02T12:00:03.913 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:03 ntpd[7524]: can't open /var/log/ntpstats/peerstats.20210302: Permission denied 2021-03-02T12:00:03.914 INFO:teuthology.orchestra.run.smithi036.stdout: 2 Mar 12:00:03 ntpd[7524]: ntpd: time slew -0.000017 s 2021-03-02T12:00:03.914 INFO:teuthology.orchestra.run.smithi036.stdout:ntpd: time slew -0.000017s 2021-03-02T12:00:03.914 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:03 ntpd[7524]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-02T12:00:03.915 INFO:teuthology.orchestra.run.smithi036.stderr: 2 Mar 12:00:03 ntpd[7524]: can't open /var/log/ntpstats/loopstats.20210302: Permission denied 2021-03-02T12:00:03.966 INFO:teuthology.orchestra.run.smithi036.stdout: remote refid st t when poll reach delay offset jitter 2021-03-02T12:00:03.967 INFO:teuthology.orchestra.run.smithi036.stdout:============================================================================== 2021-03-02T12:00:03.967 INFO:teuthology.orchestra.run.smithi036.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T12:00:03.968 INFO:teuthology.orchestra.run.smithi036.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T12:00:03.968 INFO:teuthology.orchestra.run.smithi036.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T12:00:03.968 INFO:teuthology.orchestra.run.smithi036.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-02T12:00:03.969 INFO:teuthology.run_tasks:Running task install... 2021-03-02T12:00:03.984 DEBUG:teuthology.task.install:project ceph 2021-03-02T12:00:03.985 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '3664024eebddedeee285ff2f143b16556af4e85d'}} 2021-03-02T12:00:03.986 DEBUG:teuthology.task.install:config {'sha1': '3664024eebddedeee285ff2f143b16556af4e85d'} 2021-03-02T12:00:03.986 INFO:teuthology.task.install:Using flavor: basic 2021-03-02T12:00:04.004 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-02T12:00:04.012 INFO:teuthology.task.install:extra packages: [] 2021-03-02T12:00:04.021 DEBUG:teuthology.orchestra.run.smithi012:> sudo lsb_release -is 2021-03-02T12:00:04.097 INFO:teuthology.orchestra.run.smithi012.stdout:Ubuntu 2021-03-02T12:00:04.102 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-02T12:00:04.103 DEBUG:teuthology.orchestra.run.smithi036:> sudo lsb_release -is 2021-03-02T12:00:04.104 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-key list | grep Ceph 2021-03-02T12:00:04.157 INFO:teuthology.orchestra.run.smithi036.stdout:Ubuntu 2021-03-02T12:00:04.162 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-02T12:00:04.163 DEBUG:teuthology.orchestra.run.smithi036:> sudo apt-key list | grep Ceph 2021-03-02T12:00:04.268 INFO:teuthology.orchestra.run.smithi012.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-02T12:00:04.279 INFO:teuthology.orchestra.run.smithi036.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-02T12:00:04.325 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-02T12:00:04.326 INFO:teuthology.orchestra.run.smithi012.stdout:uid [ unknown] Ceph.com (release key) 2021-03-02T12:00:04.327 DEBUG:teuthology.orchestra.run.smithi012:> sudo lsb_release -is 2021-03-02T12:00:04.339 INFO:teuthology.orchestra.run.smithi036.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-02T12:00:04.339 INFO:teuthology.orchestra.run.smithi036.stdout:uid [ unknown] Ceph.com (release key) 2021-03-02T12:00:04.340 DEBUG:teuthology.orchestra.run.smithi036:> sudo lsb_release -is 2021-03-02T12:00:04.396 INFO:teuthology.orchestra.run.smithi012.stdout:Ubuntu 2021-03-02T12:00:04.401 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-02T12:00:04.403 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-02T12:00:04.404 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3664024eebddedeee285ff2f143b16556af4e85d 2021-03-02T12:00:04.412 INFO:teuthology.orchestra.run.smithi036.stdout:Ubuntu 2021-03-02T12:00:04.413 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-02T12:00:04.413 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-02T12:00:04.413 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3664024eebddedeee285ff2f143b16556af4e85d 2021-03-02T12:00:04.587 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default/ 2021-03-02T12:00:04.588 INFO:teuthology.task.install.deb:Package version is 15.2.9-47-g3664024e-1bionic 2021-03-02T12:00:04.593 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default/ 2021-03-02T12:00:04.593 INFO:teuthology.task.install.deb:Package version is 15.2.9-47-g3664024e-1bionic 2021-03-02T12:00:04.745 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:00:04.747 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-02T12:00:04.752 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:00:04.868 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-02T12:00:04.896 DEBUG:teuthology.orchestra.run.smithi012:> sudo apt-get update 2021-03-02T12:00:04.927 DEBUG:teuthology.orchestra.run.smithi036:> sudo apt-get update 2021-03-02T12:00:05.146 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic InRelease 2021-03-02T12:00:05.169 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-02T12:00:05.188 INFO:teuthology.orchestra.run.smithi036.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic InRelease 2021-03-02T12:00:05.194 INFO:teuthology.orchestra.run.smithi012.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-02T12:00:05.203 INFO:teuthology.orchestra.run.smithi036.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic Release [7,681 B] 2021-03-02T12:00:05.218 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-03-02T12:00:05.229 INFO:teuthology.orchestra.run.smithi036.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-02T12:00:05.252 INFO:teuthology.orchestra.run.smithi036.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,073 B] 2021-03-02T12:00:05.254 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,073 B] 2021-03-02T12:00:05.288 INFO:teuthology.orchestra.run.smithi036.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-03-02T12:00:05.800 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-02T12:00:05.870 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-02T12:00:05.953 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-02T12:00:06.241 INFO:teuthology.orchestra.run.smithi036.stdout:Hit:6 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-02T12:00:06.241 INFO:teuthology.orchestra.run.smithi036.stdout:Hit:7 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-02T12:00:06.308 INFO:teuthology.orchestra.run.smithi036.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-02T12:00:06.386 INFO:teuthology.orchestra.run.smithi036.stdout:Hit:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-02T12:00:06.490 INFO:teuthology.orchestra.run.smithi012.stdout:Hit:9 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-02T12:00:06.710 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 26.6 kB in 2s (16.5 kB/s) 2021-03-02T12:00:06.875 INFO:teuthology.orchestra.run.smithi036.stdout:Fetched 26.6 kB in 2s (15.2 kB/s) 2021-03-02T12:00:08.007 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2021-03-02T12:00:08.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.9-47-g3664024e-1bionic cephadm=15.2.9-47-g3664024e-1bionic ceph-mds=15.2.9-47-g3664024e-1bionic ceph-mgr=15.2.9-47-g3664024e-1bionic ceph-common=15.2.9-47-g3664024e-1bionic ceph-fuse=15.2.9-47-g3664024e-1bionic ceph-test=15.2.9-47-g3664024e-1bionic radosgw=15.2.9-47-g3664024e-1bionic python3-rados=15.2.9-47-g3664024e-1bionic python3-rgw=15.2.9-47-g3664024e-1bionic python3-cephfs=15.2.9-47-g3664024e-1bionic python3-rbd=15.2.9-47-g3664024e-1bionic libcephfs2=15.2.9-47-g3664024e-1bionic libcephfs-dev=15.2.9-47-g3664024e-1bionic librados2=15.2.9-47-g3664024e-1bionic librbd1=15.2.9-47-g3664024e-1bionic rbd-fuse=15.2.9-47-g3664024e-1bionic 2021-03-02T12:00:08.136 INFO:teuthology.orchestra.run.smithi012.stdout:Reading package lists... 2021-03-02T12:00:08.172 INFO:teuthology.orchestra.run.smithi036.stdout:Reading package lists... 2021-03-02T12:00:08.222 DEBUG:teuthology.orchestra.run.smithi036:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.9-47-g3664024e-1bionic cephadm=15.2.9-47-g3664024e-1bionic ceph-mds=15.2.9-47-g3664024e-1bionic ceph-mgr=15.2.9-47-g3664024e-1bionic ceph-common=15.2.9-47-g3664024e-1bionic ceph-fuse=15.2.9-47-g3664024e-1bionic ceph-test=15.2.9-47-g3664024e-1bionic radosgw=15.2.9-47-g3664024e-1bionic python3-rados=15.2.9-47-g3664024e-1bionic python3-rgw=15.2.9-47-g3664024e-1bionic python3-cephfs=15.2.9-47-g3664024e-1bionic python3-rbd=15.2.9-47-g3664024e-1bionic libcephfs2=15.2.9-47-g3664024e-1bionic libcephfs-dev=15.2.9-47-g3664024e-1bionic librados2=15.2.9-47-g3664024e-1bionic librbd1=15.2.9-47-g3664024e-1bionic rbd-fuse=15.2.9-47-g3664024e-1bionic 2021-03-02T12:00:08.302 INFO:teuthology.orchestra.run.smithi036.stdout:Reading package lists... 2021-03-02T12:00:08.407 INFO:teuthology.orchestra.run.smithi012.stdout:Building dependency tree... 2021-03-02T12:00:08.408 INFO:teuthology.orchestra.run.smithi012.stdout:Reading state information... 2021-03-02T12:00:08.579 INFO:teuthology.orchestra.run.smithi036.stdout:Building dependency tree... 2021-03-02T12:00:08.580 INFO:teuthology.orchestra.run.smithi036.stdout:Reading state information... 2021-03-02T12:00:08.586 INFO:teuthology.orchestra.run.smithi012.stdout:The following additional packages will be installed: 2021-03-02T12:00:08.586 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-02T12:00:08.587 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-02T12:00:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-03-02T12:00:08.588 INFO:teuthology.orchestra.run.smithi012.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-02T12:00:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-02T12:00:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-02T12:00:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-02T12:00:08.589 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-02T12:00:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-02T12:00:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-02T12:00:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-02T12:00:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-02T12:00:08.590 INFO:teuthology.orchestra.run.smithi012.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-02T12:00:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-02T12:00:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-02T12:00:08.591 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-02T12:00:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-02T12:00:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-02T12:00:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-02T12:00:08.592 INFO:teuthology.orchestra.run.smithi012.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-02T12:00:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-02T12:00:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-02T12:00:08.593 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-02T12:00:08.594 INFO:teuthology.orchestra.run.smithi012.stdout:Suggested packages: 2021-03-02T12:00:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-02T12:00:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-02T12:00:08.594 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-02T12:00:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-02T12:00:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-02T12:00:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-02T12:00:08.595 INFO:teuthology.orchestra.run.smithi012.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-02T12:00:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-02T12:00:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-02T12:00:08.596 INFO:teuthology.orchestra.run.smithi012.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-02T12:00:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-02T12:00:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-02T12:00:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-02T12:00:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-02T12:00:08.597 INFO:teuthology.orchestra.run.smithi012.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-02T12:00:08.700 INFO:teuthology.orchestra.run.smithi012.stdout:The following NEW packages will be installed: 2021-03-02T12:00:08.700 INFO:teuthology.orchestra.run.smithi012.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-02T12:00:08.701 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-02T12:00:08.701 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-02T12:00:08.701 INFO:teuthology.orchestra.run.smithi012.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-03-02T12:00:08.703 INFO:teuthology.orchestra.run.smithi012.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-02T12:00:08.703 INFO:teuthology.orchestra.run.smithi012.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-02T12:00:08.703 INFO:teuthology.orchestra.run.smithi012.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-02T12:00:08.704 INFO:teuthology.orchestra.run.smithi012.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-02T12:00:08.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-02T12:00:08.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-02T12:00:08.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-02T12:00:08.704 INFO:teuthology.orchestra.run.smithi012.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-02T12:00:08.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-02T12:00:08.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-02T12:00:08.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-02T12:00:08.705 INFO:teuthology.orchestra.run.smithi012.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-02T12:00:08.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-02T12:00:08.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-02T12:00:08.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-02T12:00:08.706 INFO:teuthology.orchestra.run.smithi012.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-02T12:00:08.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-02T12:00:08.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-02T12:00:08.707 INFO:teuthology.orchestra.run.smithi012.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-02T12:00:08.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-02T12:00:08.708 INFO:teuthology.orchestra.run.smithi012.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-02T12:00:08.708 INFO:teuthology.orchestra.run.smithi012.stdout:The following packages will be upgraded: 2021-03-02T12:00:08.709 INFO:teuthology.orchestra.run.smithi012.stdout: librados2 librbd1 2021-03-02T12:00:08.767 INFO:teuthology.orchestra.run.smithi036.stdout:The following additional packages will be installed: 2021-03-02T12:00:08.768 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-02T12:00:08.768 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-02T12:00:08.769 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-03-02T12:00:08.770 INFO:teuthology.orchestra.run.smithi036.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-02T12:00:08.770 INFO:teuthology.orchestra.run.smithi036.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-02T12:00:08.770 INFO:teuthology.orchestra.run.smithi036.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-02T12:00:08.771 INFO:teuthology.orchestra.run.smithi036.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-02T12:00:08.771 INFO:teuthology.orchestra.run.smithi036.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-02T12:00:08.771 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-02T12:00:08.771 INFO:teuthology.orchestra.run.smithi036.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-02T12:00:08.772 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-02T12:00:08.772 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-02T12:00:08.772 INFO:teuthology.orchestra.run.smithi036.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-02T12:00:08.772 INFO:teuthology.orchestra.run.smithi036.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-02T12:00:08.773 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-02T12:00:08.773 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-02T12:00:08.773 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-02T12:00:08.773 INFO:teuthology.orchestra.run.smithi036.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-02T12:00:08.774 INFO:teuthology.orchestra.run.smithi036.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-02T12:00:08.774 INFO:teuthology.orchestra.run.smithi036.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-02T12:00:08.774 INFO:teuthology.orchestra.run.smithi036.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-02T12:00:08.774 INFO:teuthology.orchestra.run.smithi036.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-02T12:00:08.775 INFO:teuthology.orchestra.run.smithi036.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-02T12:00:08.775 INFO:teuthology.orchestra.run.smithi036.stdout:Suggested packages: 2021-03-02T12:00:08.775 INFO:teuthology.orchestra.run.smithi036.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-02T12:00:08.776 INFO:teuthology.orchestra.run.smithi036.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-02T12:00:08.776 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-02T12:00:08.783 INFO:teuthology.orchestra.run.smithi036.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-02T12:00:08.783 INFO:teuthology.orchestra.run.smithi036.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-02T12:00:08.783 INFO:teuthology.orchestra.run.smithi036.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-02T12:00:08.784 INFO:teuthology.orchestra.run.smithi036.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-02T12:00:08.784 INFO:teuthology.orchestra.run.smithi036.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-02T12:00:08.784 INFO:teuthology.orchestra.run.smithi036.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-02T12:00:08.785 INFO:teuthology.orchestra.run.smithi036.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-02T12:00:08.785 INFO:teuthology.orchestra.run.smithi036.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-02T12:00:08.785 INFO:teuthology.orchestra.run.smithi036.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-02T12:00:08.785 INFO:teuthology.orchestra.run.smithi036.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-02T12:00:08.786 INFO:teuthology.orchestra.run.smithi036.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-02T12:00:08.786 INFO:teuthology.orchestra.run.smithi036.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-02T12:00:08.866 INFO:teuthology.orchestra.run.smithi012.stdout:2 upgraded, 111 newly installed, 0 to remove and 5 not upgraded. 2021-03-02T12:00:08.866 INFO:teuthology.orchestra.run.smithi012.stdout:Need to get 156 MB of archives. 2021-03-02T12:00:08.867 INFO:teuthology.orchestra.run.smithi012.stdout:After this operation, 725 MB of additional disk space will be used. 2021-03-02T12:00:08.867 INFO:teuthology.orchestra.run.smithi012.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.9-47-g3664024e-1bionic [1,638 kB] 2021-03-02T12:00:08.877 INFO:teuthology.orchestra.run.smithi036.stdout:The following NEW packages will be installed: 2021-03-02T12:00:08.878 INFO:teuthology.orchestra.run.smithi036.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-02T12:00:08.878 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-02T12:00:08.878 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-02T12:00:08.880 INFO:teuthology.orchestra.run.smithi036.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-03-02T12:00:08.880 INFO:teuthology.orchestra.run.smithi036.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-02T12:00:08.881 INFO:teuthology.orchestra.run.smithi036.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-02T12:00:08.881 INFO:teuthology.orchestra.run.smithi036.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-02T12:00:08.881 INFO:teuthology.orchestra.run.smithi036.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-02T12:00:08.881 INFO:teuthology.orchestra.run.smithi036.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-02T12:00:08.882 INFO:teuthology.orchestra.run.smithi036.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-02T12:00:08.882 INFO:teuthology.orchestra.run.smithi036.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-02T12:00:08.882 INFO:teuthology.orchestra.run.smithi036.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-02T12:00:08.882 INFO:teuthology.orchestra.run.smithi036.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-02T12:00:08.882 INFO:teuthology.orchestra.run.smithi036.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-02T12:00:08.883 INFO:teuthology.orchestra.run.smithi036.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-02T12:00:08.883 INFO:teuthology.orchestra.run.smithi036.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-02T12:00:08.883 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-02T12:00:08.884 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-02T12:00:08.884 INFO:teuthology.orchestra.run.smithi036.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-02T12:00:08.884 INFO:teuthology.orchestra.run.smithi036.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-02T12:00:08.884 INFO:teuthology.orchestra.run.smithi036.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-02T12:00:08.885 INFO:teuthology.orchestra.run.smithi036.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-02T12:00:08.885 INFO:teuthology.orchestra.run.smithi036.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-02T12:00:08.885 INFO:teuthology.orchestra.run.smithi036.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-02T12:00:08.886 INFO:teuthology.orchestra.run.smithi036.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-02T12:00:08.886 INFO:teuthology.orchestra.run.smithi036.stdout:The following packages will be upgraded: 2021-03-02T12:00:08.887 INFO:teuthology.orchestra.run.smithi036.stdout: librados2 librbd1 2021-03-02T12:00:08.893 INFO:teuthology.orchestra.run.smithi012.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-02T12:00:08.974 INFO:teuthology.orchestra.run.smithi012.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-02T12:00:09.040 INFO:teuthology.orchestra.run.smithi012.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.9-47-g3664024e-1bionic [3,059 kB] 2021-03-02T12:00:09.068 INFO:teuthology.orchestra.run.smithi036.stdout:2 upgraded, 111 newly installed, 0 to remove and 5 not upgraded. 2021-03-02T12:00:09.069 INFO:teuthology.orchestra.run.smithi036.stdout:Need to get 156 MB of archives. 2021-03-02T12:00:09.069 INFO:teuthology.orchestra.run.smithi036.stdout:After this operation, 725 MB of additional disk space will be used. 2021-03-02T12:00:09.069 INFO:teuthology.orchestra.run.smithi036.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-02T12:00:09.075 INFO:teuthology.orchestra.run.smithi036.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.9-47-g3664024e-1bionic [1,638 kB] 2021-03-02T12:00:09.135 INFO:teuthology.orchestra.run.smithi012.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.9-47-g3664024e-1bionic [436 kB] 2021-03-02T12:00:09.147 INFO:teuthology.orchestra.run.smithi036.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-02T12:00:09.148 INFO:teuthology.orchestra.run.smithi012.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.9-47-g3664024e-1bionic [345 kB] 2021-03-02T12:00:09.159 INFO:teuthology.orchestra.run.smithi012.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.9-47-g3664024e-1bionic [20.7 kB] 2021-03-02T12:00:09.160 INFO:teuthology.orchestra.run.smithi012.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.9-47-g3664024e-1bionic [133 kB] 2021-03-02T12:00:09.164 INFO:teuthology.orchestra.run.smithi012.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.9-47-g3664024e-1bionic [29.0 kB] 2021-03-02T12:00:09.165 INFO:teuthology.orchestra.run.smithi012.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.9-47-g3664024e-1bionic [325 kB] 2021-03-02T12:00:09.176 INFO:teuthology.orchestra.run.smithi012.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.9-47-g3664024e-1bionic [2,904 kB] 2021-03-02T12:00:09.222 INFO:teuthology.orchestra.run.smithi012.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-02T12:00:09.229 INFO:teuthology.orchestra.run.smithi012.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-02T12:00:09.231 INFO:teuthology.orchestra.run.smithi012.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-02T12:00:09.278 INFO:teuthology.orchestra.run.smithi012.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.9-47-g3664024e-1bionic [99.6 kB] 2021-03-02T12:00:09.280 INFO:teuthology.orchestra.run.smithi012.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.9-47-g3664024e-1bionic [339 kB] 2021-03-02T12:00:09.292 INFO:teuthology.orchestra.run.smithi012.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.9-47-g3664024e-1bionic [18.8 MB] 2021-03-02T12:00:09.344 INFO:teuthology.orchestra.run.smithi012.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-02T12:00:09.347 INFO:teuthology.orchestra.run.smithi012.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-02T12:00:09.384 INFO:teuthology.orchestra.run.smithi012.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-02T12:00:09.392 INFO:teuthology.orchestra.run.smithi012.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-02T12:00:09.394 INFO:teuthology.orchestra.run.smithi036.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-02T12:00:09.399 INFO:teuthology.orchestra.run.smithi012.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-02T12:00:09.405 INFO:teuthology.orchestra.run.smithi036.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-02T12:00:09.409 INFO:teuthology.orchestra.run.smithi036.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-02T12:00:09.410 INFO:teuthology.orchestra.run.smithi012.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-02T12:00:09.420 INFO:teuthology.orchestra.run.smithi012.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-02T12:00:09.449 INFO:teuthology.orchestra.run.smithi012.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-02T12:00:09.457 INFO:teuthology.orchestra.run.smithi012.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-02T12:00:09.462 INFO:teuthology.orchestra.run.smithi012.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-02T12:00:09.478 INFO:teuthology.orchestra.run.smithi012.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-02T12:00:09.513 INFO:teuthology.orchestra.run.smithi036.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-02T12:00:09.943 INFO:teuthology.orchestra.run.smithi012.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-02T12:00:09.944 INFO:teuthology.orchestra.run.smithi012.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-02T12:00:09.944 INFO:teuthology.orchestra.run.smithi012.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-02T12:00:09.945 INFO:teuthology.orchestra.run.smithi012.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-02T12:00:09.945 INFO:teuthology.orchestra.run.smithi012.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-02T12:00:09.946 INFO:teuthology.orchestra.run.smithi012.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-02T12:00:09.947 INFO:teuthology.orchestra.run.smithi012.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-02T12:00:09.947 INFO:teuthology.orchestra.run.smithi012.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-02T12:00:09.947 INFO:teuthology.orchestra.run.smithi012.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-02T12:00:09.948 INFO:teuthology.orchestra.run.smithi012.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-02T12:00:09.948 INFO:teuthology.orchestra.run.smithi012.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-02T12:00:09.949 INFO:teuthology.orchestra.run.smithi012.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-02T12:00:09.949 INFO:teuthology.orchestra.run.smithi012.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-02T12:00:09.950 INFO:teuthology.orchestra.run.smithi012.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-02T12:00:09.950 INFO:teuthology.orchestra.run.smithi012.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-02T12:00:09.951 INFO:teuthology.orchestra.run.smithi012.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-02T12:00:09.951 INFO:teuthology.orchestra.run.smithi012.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-02T12:00:09.951 INFO:teuthology.orchestra.run.smithi012.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-02T12:00:09.952 INFO:teuthology.orchestra.run.smithi012.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-02T12:00:09.952 INFO:teuthology.orchestra.run.smithi012.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-02T12:00:09.952 INFO:teuthology.orchestra.run.smithi012.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-02T12:00:09.954 INFO:teuthology.orchestra.run.smithi036.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-02T12:00:09.954 INFO:teuthology.orchestra.run.smithi036.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-02T12:00:09.954 INFO:teuthology.orchestra.run.smithi036.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-02T12:00:09.955 INFO:teuthology.orchestra.run.smithi036.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-02T12:00:09.955 INFO:teuthology.orchestra.run.smithi036.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-02T12:00:09.956 INFO:teuthology.orchestra.run.smithi036.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-02T12:00:09.956 INFO:teuthology.orchestra.run.smithi036.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-02T12:00:09.956 INFO:teuthology.orchestra.run.smithi036.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-02T12:00:09.956 INFO:teuthology.orchestra.run.smithi036.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-02T12:00:09.956 INFO:teuthology.orchestra.run.smithi036.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-02T12:00:09.957 INFO:teuthology.orchestra.run.smithi036.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-02T12:00:09.957 INFO:teuthology.orchestra.run.smithi036.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-02T12:00:09.957 INFO:teuthology.orchestra.run.smithi036.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-02T12:00:09.957 INFO:teuthology.orchestra.run.smithi036.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-02T12:00:09.958 INFO:teuthology.orchestra.run.smithi036.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-02T12:00:09.958 INFO:teuthology.orchestra.run.smithi036.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-02T12:00:09.958 INFO:teuthology.orchestra.run.smithi036.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-02T12:00:09.958 INFO:teuthology.orchestra.run.smithi036.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-02T12:00:09.958 INFO:teuthology.orchestra.run.smithi036.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-02T12:00:09.959 INFO:teuthology.orchestra.run.smithi036.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-02T12:00:09.959 INFO:teuthology.orchestra.run.smithi036.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-02T12:00:09.959 INFO:teuthology.orchestra.run.smithi036.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-02T12:00:09.959 INFO:teuthology.orchestra.run.smithi036.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-02T12:00:09.959 INFO:teuthology.orchestra.run.smithi036.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-02T12:00:09.960 INFO:teuthology.orchestra.run.smithi036.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-02T12:00:09.960 INFO:teuthology.orchestra.run.smithi036.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-02T12:00:09.960 INFO:teuthology.orchestra.run.smithi036.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-02T12:00:09.960 INFO:teuthology.orchestra.run.smithi036.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-02T12:00:09.961 INFO:teuthology.orchestra.run.smithi036.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-02T12:00:09.961 INFO:teuthology.orchestra.run.smithi036.stdout:Get:37 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.9-47-g3664024e-1bionic [3,059 kB] 2021-03-02T12:00:09.962 INFO:teuthology.orchestra.run.smithi012.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.9-47-g3664024e-1bionic [5,175 kB] 2021-03-02T12:00:10.024 INFO:teuthology.orchestra.run.smithi036.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-02T12:00:10.024 INFO:teuthology.orchestra.run.smithi036.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-02T12:00:10.148 INFO:teuthology.orchestra.run.smithi012.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.9-47-g3664024e-1bionic [154 kB] 2021-03-02T12:00:10.275 INFO:teuthology.orchestra.run.smithi012.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-02T12:00:10.275 INFO:teuthology.orchestra.run.smithi012.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.9-47-g3664024e-1bionic [1,285 kB] 2021-03-02T12:00:10.275 INFO:teuthology.orchestra.run.smithi012.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-02T12:00:10.276 INFO:teuthology.orchestra.run.smithi012.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-02T12:00:10.276 INFO:teuthology.orchestra.run.smithi012.stdout:Get:56 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.9-47-g3664024e-1bionic [5,943 kB] 2021-03-02T12:00:10.276 INFO:teuthology.orchestra.run.smithi012.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-02T12:00:10.276 INFO:teuthology.orchestra.run.smithi012.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-02T12:00:10.277 INFO:teuthology.orchestra.run.smithi012.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-02T12:00:10.277 INFO:teuthology.orchestra.run.smithi012.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-02T12:00:10.277 INFO:teuthology.orchestra.run.smithi012.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-02T12:00:10.278 INFO:teuthology.orchestra.run.smithi012.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-02T12:00:10.278 INFO:teuthology.orchestra.run.smithi012.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-02T12:00:10.303 INFO:teuthology.orchestra.run.smithi036.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-02T12:00:10.313 INFO:teuthology.orchestra.run.smithi012.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-02T12:00:10.315 INFO:teuthology.orchestra.run.smithi012.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-02T12:00:10.339 INFO:teuthology.orchestra.run.smithi036.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-02T12:00:10.342 INFO:teuthology.orchestra.run.smithi036.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-02T12:00:10.382 INFO:teuthology.orchestra.run.smithi036.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-02T12:00:10.382 INFO:teuthology.orchestra.run.smithi036.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-02T12:00:10.383 INFO:teuthology.orchestra.run.smithi036.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-02T12:00:10.384 INFO:teuthology.orchestra.run.smithi036.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-02T12:00:10.385 INFO:teuthology.orchestra.run.smithi036.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-02T12:00:10.386 INFO:teuthology.orchestra.run.smithi036.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-02T12:00:10.387 INFO:teuthology.orchestra.run.smithi036.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-02T12:00:10.392 INFO:teuthology.orchestra.run.smithi012.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-02T12:00:10.395 INFO:teuthology.orchestra.run.smithi012.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-02T12:00:10.400 INFO:teuthology.orchestra.run.smithi012.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-02T12:00:10.400 INFO:teuthology.orchestra.run.smithi012.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-02T12:00:10.402 INFO:teuthology.orchestra.run.smithi012.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-02T12:00:10.512 INFO:teuthology.orchestra.run.smithi036.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-02T12:00:10.512 INFO:teuthology.orchestra.run.smithi036.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-02T12:00:10.513 INFO:teuthology.orchestra.run.smithi012.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-02T12:00:10.513 INFO:teuthology.orchestra.run.smithi012.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-02T12:00:10.514 INFO:teuthology.orchestra.run.smithi012.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-02T12:00:10.514 INFO:teuthology.orchestra.run.smithi012.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.9-47-g3664024e-1bionic [22.8 MB] 2021-03-02T12:00:10.515 INFO:teuthology.orchestra.run.smithi012.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-02T12:00:10.515 INFO:teuthology.orchestra.run.smithi012.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-02T12:00:10.539 INFO:teuthology.orchestra.run.smithi036.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-02T12:00:10.540 INFO:teuthology.orchestra.run.smithi036.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-02T12:00:10.546 INFO:teuthology.orchestra.run.smithi036.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-02T12:00:10.547 INFO:teuthology.orchestra.run.smithi036.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-02T12:00:10.547 INFO:teuthology.orchestra.run.smithi036.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-02T12:00:10.549 INFO:teuthology.orchestra.run.smithi036.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-02T12:00:10.551 INFO:teuthology.orchestra.run.smithi036.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-02T12:00:10.551 INFO:teuthology.orchestra.run.smithi012.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-02T12:00:10.552 INFO:teuthology.orchestra.run.smithi036.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-02T12:00:10.558 INFO:teuthology.orchestra.run.smithi012.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-02T12:00:10.559 INFO:teuthology.orchestra.run.smithi012.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-02T12:00:10.560 INFO:teuthology.orchestra.run.smithi012.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-02T12:00:10.561 INFO:teuthology.orchestra.run.smithi012.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-02T12:00:10.562 INFO:teuthology.orchestra.run.smithi012.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-02T12:00:10.564 INFO:teuthology.orchestra.run.smithi012.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-02T12:00:10.564 INFO:teuthology.orchestra.run.smithi012.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-02T12:00:10.617 INFO:teuthology.orchestra.run.smithi036.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-02T12:00:10.622 INFO:teuthology.orchestra.run.smithi036.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-02T12:00:10.630 INFO:teuthology.orchestra.run.smithi012.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-02T12:00:10.631 INFO:teuthology.orchestra.run.smithi012.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-02T12:00:10.695 INFO:teuthology.orchestra.run.smithi036.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-02T12:00:10.703 INFO:teuthology.orchestra.run.smithi036.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-02T12:00:10.704 INFO:teuthology.orchestra.run.smithi036.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-02T12:00:10.704 INFO:teuthology.orchestra.run.smithi036.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-02T12:00:10.705 INFO:teuthology.orchestra.run.smithi036.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-02T12:00:10.707 INFO:teuthology.orchestra.run.smithi036.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-02T12:00:10.709 INFO:teuthology.orchestra.run.smithi012.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-02T12:00:10.710 INFO:teuthology.orchestra.run.smithi036.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-02T12:00:10.711 INFO:teuthology.orchestra.run.smithi036.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-02T12:00:10.732 INFO:teuthology.orchestra.run.smithi012.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-02T12:00:10.733 INFO:teuthology.orchestra.run.smithi012.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-02T12:00:10.733 INFO:teuthology.orchestra.run.smithi012.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-02T12:00:10.772 INFO:teuthology.orchestra.run.smithi036.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.9-47-g3664024e-1bionic [436 kB] 2021-03-02T12:00:10.774 INFO:teuthology.orchestra.run.smithi036.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-02T12:00:10.774 INFO:teuthology.orchestra.run.smithi036.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-02T12:00:10.847 INFO:teuthology.orchestra.run.smithi036.stdout:Get:73 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.9-47-g3664024e-1bionic [345 kB] 2021-03-02T12:00:10.852 INFO:teuthology.orchestra.run.smithi036.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-02T12:00:10.871 INFO:teuthology.orchestra.run.smithi012.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-02T12:00:10.872 INFO:teuthology.orchestra.run.smithi012.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-02T12:00:10.872 INFO:teuthology.orchestra.run.smithi012.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-02T12:00:10.875 INFO:teuthology.orchestra.run.smithi036.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-02T12:00:10.876 INFO:teuthology.orchestra.run.smithi036.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-02T12:00:10.877 INFO:teuthology.orchestra.run.smithi036.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-02T12:00:10.878 INFO:teuthology.orchestra.run.smithi012.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-02T12:00:10.878 INFO:teuthology.orchestra.run.smithi012.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-02T12:00:10.879 INFO:teuthology.orchestra.run.smithi012.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-02T12:00:10.879 INFO:teuthology.orchestra.run.smithi012.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-02T12:00:10.880 INFO:teuthology.orchestra.run.smithi012.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-02T12:00:10.881 INFO:teuthology.orchestra.run.smithi012.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-02T12:00:10.921 INFO:teuthology.orchestra.run.smithi036.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.9-47-g3664024e-1bionic [20.7 kB] 2021-03-02T12:00:10.922 INFO:teuthology.orchestra.run.smithi036.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.9-47-g3664024e-1bionic [133 kB] 2021-03-02T12:00:10.970 INFO:teuthology.orchestra.run.smithi036.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.9-47-g3664024e-1bionic [29.0 kB] 2021-03-02T12:00:10.988 INFO:teuthology.orchestra.run.smithi036.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.9-47-g3664024e-1bionic [325 kB] 2021-03-02T12:00:10.999 INFO:teuthology.orchestra.run.smithi036.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.9-47-g3664024e-1bionic [2,904 kB] 2021-03-02T12:00:11.016 INFO:teuthology.orchestra.run.smithi036.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-02T12:00:11.218 INFO:teuthology.orchestra.run.smithi036.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-02T12:00:11.219 INFO:teuthology.orchestra.run.smithi036.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.4 [329 kB] 2021-03-02T12:00:11.219 INFO:teuthology.orchestra.run.smithi036.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-02T12:00:11.220 INFO:teuthology.orchestra.run.smithi036.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-02T12:00:11.220 INFO:teuthology.orchestra.run.smithi036.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-02T12:00:11.222 INFO:teuthology.orchestra.run.smithi036.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-02T12:00:11.222 INFO:teuthology.orchestra.run.smithi036.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-02T12:00:11.223 INFO:teuthology.orchestra.run.smithi036.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-02T12:00:11.323 INFO:teuthology.orchestra.run.smithi012.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.9-47-g3664024e-1bionic [3,932 B] 2021-03-02T12:00:11.533 INFO:teuthology.orchestra.run.smithi012.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.9-47-g3664024e-1bionic [516 kB] 2021-03-02T12:00:11.535 INFO:teuthology.orchestra.run.smithi012.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.9-47-g3664024e-1bionic [1,854 kB] 2021-03-02T12:00:11.536 INFO:teuthology.orchestra.run.smithi012.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.9-47-g3664024e-1bionic [52.2 kB] 2021-03-02T12:00:11.536 INFO:teuthology.orchestra.run.smithi012.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.9-47-g3664024e-1bionic [57.9 kB] 2021-03-02T12:00:11.537 INFO:teuthology.orchestra.run.smithi012.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.9-47-g3664024e-1bionic [3,196 kB] 2021-03-02T12:00:11.545 INFO:teuthology.orchestra.run.smithi036.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.9-47-g3664024e-1bionic [99.6 kB] 2021-03-02T12:00:11.600 INFO:teuthology.orchestra.run.smithi012.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.9-47-g3664024e-1bionic [34.3 kB] 2021-03-02T12:00:11.601 INFO:teuthology.orchestra.run.smithi012.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.9-47-g3664024e-1bionic [7,890 kB] 2021-03-02T12:00:11.661 INFO:teuthology.orchestra.run.smithi036.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.9-47-g3664024e-1bionic [339 kB] 2021-03-02T12:00:11.884 INFO:teuthology.orchestra.run.smithi036.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.9-47-g3664024e-1bionic [18.8 MB] 2021-03-02T12:00:11.885 INFO:teuthology.orchestra.run.smithi012.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.9-47-g3664024e-1bionic [4,044 B] 2021-03-02T12:00:11.885 INFO:teuthology.orchestra.run.smithi012.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.9-47-g3664024e-1bionic [22.5 kB] 2021-03-02T12:00:11.885 INFO:teuthology.orchestra.run.smithi012.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.9-47-g3664024e-1bionic [44.1 MB] 2021-03-02T12:00:13.775 INFO:teuthology.orchestra.run.smithi012.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.9-47-g3664024e-1bionic [18.0 kB] 2021-03-02T12:00:13.775 INFO:teuthology.orchestra.run.smithi012.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.9-47-g3664024e-1bionic [8,877 kB] 2021-03-02T12:00:14.146 INFO:teuthology.orchestra.run.smithi012.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.9-47-g3664024e-1bionic [63.2 kB] 2021-03-02T12:00:14.182 INFO:teuthology.orchestra.run.smithi036.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.9-47-g3664024e-1bionic [5,175 kB] 2021-03-02T12:00:14.412 INFO:teuthology.orchestra.run.smithi036.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.9-47-g3664024e-1bionic [154 kB] 2021-03-02T12:00:14.423 INFO:teuthology.orchestra.run.smithi036.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.9-47-g3664024e-1bionic [1,285 kB] 2021-03-02T12:00:14.476 INFO:teuthology.orchestra.run.smithi036.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.9-47-g3664024e-1bionic [5,943 kB] 2021-03-02T12:00:14.865 INFO:teuthology.orchestra.run.smithi012.stdout:Fetched 156 MB in 5s (28.8 MB/s) 2021-03-02T12:00:15.010 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-02T12:00:16.736 INFO:teuthology.orchestra.run.smithi036.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.9-47-g3664024e-1bionic [22.8 MB] 2021-03-02T12:00:16.739 INFO:teuthology.orchestra.run.smithi012.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-02T12:00:16.739 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-02T12:00:16.739 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-02T12:00:16.739 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-02T12:00:16.740 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-02T12:00:16.740 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-02T12:00:16.740 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../002-librbd1_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:16.740 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librbd1 (15.2.9-47-g3664024e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-02T12:00:16.741 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../003-librados2_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:16.741 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librados2 (15.2.9-47-g3664024e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-02T12:00:17.281 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs2. 2021-03-02T12:00:17.293 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../004-libcephfs2_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:17.347 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:17.642 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rados. 2021-03-02T12:00:17.653 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../005-python3-rados_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:17.692 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rados (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:18.194 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-02T12:00:18.206 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:18.245 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-argparse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:18.506 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cephfs. 2021-03-02T12:00:18.518 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../007-python3-cephfs_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:18.555 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cephfs (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:18.846 INFO:teuthology.orchestra.run.smithi036.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.9-47-g3664024e-1bionic [3,932 B] 2021-03-02T12:00:18.846 INFO:teuthology.orchestra.run.smithi036.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.9-47-g3664024e-1bionic [516 kB] 2021-03-02T12:00:18.891 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-02T12:00:18.903 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:18.922 INFO:teuthology.orchestra.run.smithi036.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.9-47-g3664024e-1bionic [1,854 kB] 2021-03-02T12:00:18.941 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:19.241 INFO:teuthology.orchestra.run.smithi036.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.9-47-g3664024e-1bionic [52.2 kB] 2021-03-02T12:00:19.269 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-prettytable. 2021-03-02T12:00:19.280 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-03-02T12:00:19.296 INFO:teuthology.orchestra.run.smithi036.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.9-47-g3664024e-1bionic [57.9 kB] 2021-03-02T12:00:19.301 INFO:teuthology.orchestra.run.smithi036.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.9-47-g3664024e-1bionic [3,196 kB] 2021-03-02T12:00:19.319 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-02T12:00:19.571 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rbd. 2021-03-02T12:00:19.582 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../010-python3-rbd_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:19.621 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rbd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:19.860 INFO:teuthology.orchestra.run.smithi036.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.9-47-g3664024e-1bionic [34.3 kB] 2021-03-02T12:00:19.861 INFO:teuthology.orchestra.run.smithi036.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.9-47-g3664024e-1bionic [7,890 kB] 2021-03-02T12:00:19.940 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-02T12:00:19.951 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-02T12:00:20.007 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-02T12:00:20.344 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-02T12:00:20.356 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-02T12:00:20.393 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-02T12:00:20.462 INFO:teuthology.orchestra.run.smithi036.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.9-47-g3664024e-1bionic [4,044 B] 2021-03-02T12:00:20.462 INFO:teuthology.orchestra.run.smithi036.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.9-47-g3664024e-1bionic [22.5 kB] 2021-03-02T12:00:20.465 INFO:teuthology.orchestra.run.smithi036.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.9-47-g3664024e-1bionic [44.1 MB] 2021-03-02T12:00:20.729 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package librgw2. 2021-03-02T12:00:20.749 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../013-librgw2_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:20.779 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking librgw2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:21.323 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rgw. 2021-03-02T12:00:21.337 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../014-python3-rgw_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:21.365 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:21.675 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-02T12:00:21.678 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-02T12:00:21.726 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-02T12:00:22.145 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-02T12:00:22.156 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-02T12:00:22.195 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-02T12:00:22.588 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package liboath0. 2021-03-02T12:00:22.600 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-03-02T12:00:22.647 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-02T12:00:22.653 INFO:teuthology.orchestra.run.smithi036.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.9-47-g3664024e-1bionic [18.0 kB] 2021-03-02T12:00:22.811 INFO:teuthology.orchestra.run.smithi036.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.9-47-g3664024e-1bionic [8,877 kB] 2021-03-02T12:00:22.908 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libradosstriper1. 2021-03-02T12:00:22.921 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../018-libradosstriper1_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:22.958 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libradosstriper1 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:23.518 INFO:teuthology.orchestra.run.smithi036.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-yuri-testing-2021-03-01-1125-octopus/3664024eebddedeee285ff2f143b16556af4e85d/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.9-47-g3664024e-1bionic [63.2 kB] 2021-03-02T12:00:23.519 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-common. 2021-03-02T12:00:23.519 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../019-ceph-common_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:23.520 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:23.788 INFO:teuthology.orchestra.run.smithi036.stdout:Fetched 156 MB in 14s (11.0 MB/s) 2021-03-02T12:00:23.912 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-02T12:00:24.099 INFO:teuthology.orchestra.run.smithi036.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-02T12:00:24.099 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-02T12:00:24.100 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-02T12:00:24.305 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-02T12:00:24.316 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-02T12:00:24.397 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-02T12:00:24.720 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../002-librbd1_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:24.851 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking librbd1 (15.2.9-47-g3664024e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-02T12:00:25.260 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-base. 2021-03-02T12:00:25.270 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../020-ceph-base_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:25.308 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-base (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:25.350 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../003-librados2_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:25.455 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking librados2 (15.2.9-47-g3664024e-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-02T12:00:26.075 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libcephfs2. 2021-03-02T12:00:26.088 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../004-libcephfs2_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:26.150 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libcephfs2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:26.179 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-dateutil. 2021-03-02T12:00:26.191 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-03-02T12:00:26.229 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-02T12:00:26.452 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-rados. 2021-03-02T12:00:26.463 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../005-python3-rados_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:26.502 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-rados (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:26.581 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-02T12:00:26.593 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-02T12:00:26.631 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-02T12:00:26.838 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-02T12:00:26.849 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:26.888 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-ceph-argparse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:26.901 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-02T12:00:26.914 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-02T12:00:26.951 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-02T12:00:27.157 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-cephfs. 2021-03-02T12:00:27.169 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../007-python3-cephfs_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:27.198 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-cephfs (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:27.237 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cryptography. 2021-03-02T12:00:27.247 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-02T12:00:27.279 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-02T12:00:27.518 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-02T12:00:27.528 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:27.567 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:27.615 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-openssl. 2021-03-02T12:00:27.626 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-02T12:00:27.656 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-02T12:00:27.879 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-prettytable. 2021-03-02T12:00:27.891 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-03-02T12:00:27.928 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-02T12:00:27.959 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-02T12:00:27.972 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:28.009 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-modules-core (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:28.181 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-rbd. 2021-03-02T12:00:28.193 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../010-python3-rbd_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:28.231 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-rbd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:28.355 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-02T12:00:28.366 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-02T12:00:28.396 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-02T12:00:28.567 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-02T12:00:28.580 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-02T12:00:28.608 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-02T12:00:28.699 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-02T12:00:28.710 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-02T12:00:28.749 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-02T12:00:28.953 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-02T12:00:28.964 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-02T12:00:29.003 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-02T12:00:29.077 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jwt. 2021-03-02T12:00:29.090 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-02T12:00:29.127 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-02T12:00:29.339 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package librgw2. 2021-03-02T12:00:29.351 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../013-librgw2_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:29.388 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking librgw2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:29.455 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-logutils. 2021-03-02T12:00:29.466 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-03-02T12:00:29.521 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-02T12:00:29.832 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-02T12:00:29.845 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-02T12:00:29.882 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-02T12:00:29.916 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-rgw. 2021-03-02T12:00:29.927 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../014-python3-rgw_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:29.982 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-rgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:30.160 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-mako. 2021-03-02T12:00:30.172 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-02T12:00:30.210 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-02T12:00:30.277 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-02T12:00:30.288 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-02T12:00:30.318 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-02T12:00:30.521 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-02T12:00:30.532 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-02T12:00:30.563 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-02T12:00:30.604 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-02T12:00:30.616 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-02T12:00:30.654 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-02T12:00:30.866 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-02T12:00:30.877 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-02T12:00:30.915 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-02T12:00:31.032 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package liboath0. 2021-03-02T12:00:31.043 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-03-02T12:00:31.082 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-02T12:00:31.217 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webob. 2021-03-02T12:00:31.229 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-02T12:00:31.267 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-02T12:00:31.509 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libradosstriper1. 2021-03-02T12:00:31.522 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../018-libradosstriper1_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:31.559 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libradosstriper1 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:31.561 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-bs4. 2021-03-02T12:00:31.573 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-03-02T12:00:31.612 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-02T12:00:31.948 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-waitress. 2021-03-02T12:00:31.973 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-common. 2021-03-02T12:00:31.973 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-03-02T12:00:31.975 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../019-ceph-common_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:31.998 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-02T12:00:32.012 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:32.317 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tempita. 2021-03-02T12:00:32.330 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-03-02T12:00:32.367 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-02T12:00:32.636 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-paste. 2021-03-02T12:00:32.649 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-02T12:00:32.686 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-02T12:00:32.998 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-02T12:00:33.010 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-02T12:00:33.048 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-02T12:00:33.320 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-02T12:00:33.330 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-02T12:00:33.367 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-02T12:00:33.652 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webtest. 2021-03-02T12:00:33.665 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-02T12:00:33.702 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-02T12:00:33.834 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-base. 2021-03-02T12:00:33.847 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../020-ceph-base_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:33.893 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-base (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:34.022 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pecan. 2021-03-02T12:00:34.035 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-03-02T12:00:34.071 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-02T12:00:34.442 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-02T12:00:34.454 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-02T12:00:34.493 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-02T12:00:34.713 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-dateutil. 2021-03-02T12:00:34.740 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-03-02T12:00:34.755 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-02T12:00:34.846 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr. 2021-03-02T12:00:34.859 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../045-ceph-mgr_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:34.897 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:35.054 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-02T12:00:35.142 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-02T12:00:35.143 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-02T12:00:35.352 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-02T12:00:35.364 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-02T12:00:35.512 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-02T12:00:35.512 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mon. 2021-03-02T12:00:35.513 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../046-ceph-mon_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:35.513 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mon (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:35.680 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-cryptography. 2021-03-02T12:00:35.693 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-02T12:00:35.730 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-02T12:00:36.099 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-openssl. 2021-03-02T12:00:36.112 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-02T12:00:36.149 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-02T12:00:36.368 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-osd. 2021-03-02T12:00:36.381 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../047-ceph-osd_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:36.418 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-02T12:00:36.422 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-osd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:36.431 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:36.468 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-modules-core (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:36.830 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-02T12:00:36.844 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-02T12:00:36.880 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-02T12:00:37.186 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-02T12:00:37.195 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-02T12:00:37.233 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-02T12:00:37.578 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-jwt. 2021-03-02T12:00:37.590 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-02T12:00:37.628 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-02T12:00:37.938 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-logutils. 2021-03-02T12:00:37.950 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-03-02T12:00:37.988 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-02T12:00:38.291 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-02T12:00:38.303 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-02T12:00:38.311 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph. 2021-03-02T12:00:38.323 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../048-ceph_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:38.341 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-02T12:00:38.377 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:38.627 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-mako. 2021-03-02T12:00:38.639 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-02T12:00:38.687 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-02T12:00:39.021 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-02T12:00:39.032 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-02T12:00:39.072 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-02T12:00:39.413 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-fuse. 2021-03-02T12:00:39.427 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../049-ceph-fuse_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:39.463 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:39.800 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mds. 2021-03-02T12:00:39.811 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../050-ceph-mds_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:39.841 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mds (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:40.082 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-02T12:00:40.095 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-02T12:00:40.132 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-02T12:00:40.329 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package cephadm. 2021-03-02T12:00:40.341 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../051-cephadm_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:40.378 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:40.441 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-webob. 2021-03-02T12:00:40.447 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-02T12:00:40.501 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-02T12:00:40.698 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jinja2. 2021-03-02T12:00:40.711 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-02T12:00:40.747 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-02T12:00:40.929 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-bs4. 2021-03-02T12:00:40.940 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-03-02T12:00:40.979 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-02T12:00:41.076 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-02T12:00:41.088 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:41.126 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:41.265 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-waitress. 2021-03-02T12:00:41.369 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-03-02T12:00:41.369 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-02T12:00:41.421 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-02T12:00:41.433 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-03-02T12:00:41.463 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-02T12:00:41.609 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-tempita. 2021-03-02T12:00:41.622 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-03-02T12:00:41.659 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-02T12:00:41.749 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-routes. 2021-03-02T12:00:41.924 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-03-02T12:00:41.924 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-02T12:00:41.962 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-paste. 2021-03-02T12:00:41.974 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-02T12:00:42.012 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-02T12:00:42.226 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-02T12:00:42.238 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:42.298 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-02T12:00:42.301 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-dashboard (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:42.311 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-02T12:00:42.347 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-02T12:00:42.674 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-02T12:00:42.674 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-02T12:00:42.699 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-02T12:00:43.002 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-webtest. 2021-03-02T12:00:43.109 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-02T12:00:43.110 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-02T12:00:43.212 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-03-02T12:00:43.225 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:43.261 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:43.380 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pecan. 2021-03-02T12:00:43.392 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-03-02T12:00:43.430 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-02T12:00:43.582 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-numpy. 2021-03-02T12:00:43.595 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-02T12:00:43.639 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-02T12:00:43.792 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-02T12:00:43.805 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-02T12:00:43.842 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-02T12:00:44.212 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr. 2021-03-02T12:00:44.225 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../045-ceph-mgr_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:44.228 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-decorator. 2021-03-02T12:00:44.242 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-02T12:00:44.262 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:44.287 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-02T12:00:44.598 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-scipy. 2021-03-02T12:00:44.610 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-02T12:00:44.648 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-02T12:00:44.715 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mon. 2021-03-02T12:00:44.727 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../046-ceph-mon_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:44.764 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mon (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:45.646 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-osd. 2021-03-02T12:00:45.658 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../047-ceph-osd_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:45.696 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-osd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:45.781 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-02T12:00:46.005 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-02T12:00:46.006 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-02T12:00:46.268 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-joblib. 2021-03-02T12:00:46.471 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-02T12:00:46.472 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-02T12:00:46.671 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sklearn. 2021-03-02T12:00:46.685 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-02T12:00:46.721 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-02T12:00:47.261 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-02T12:00:47.273 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:47.318 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:47.568 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph. 2021-03-02T12:00:47.582 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../048-ceph_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:47.626 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:47.953 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-fuse. 2021-03-02T12:00:47.966 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../049-ceph-fuse_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:48.003 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:48.390 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mds. 2021-03-02T12:00:48.402 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../050-ceph-mds_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:48.439 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mds (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:48.512 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-monotonic. 2021-03-02T12:00:48.525 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-02T12:00:48.562 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-02T12:00:48.918 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package cephadm. 2021-03-02T12:00:48.930 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../051-cephadm_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:48.967 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:49.054 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-fasteners. 2021-03-02T12:00:49.062 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-02T12:00:49.106 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-02T12:00:49.294 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-jinja2. 2021-03-02T12:00:49.306 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-02T12:00:49.344 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-02T12:00:49.351 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-httplib2. 2021-03-02T12:00:49.364 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-02T12:00:49.393 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-02T12:00:49.656 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-02T12:00:49.667 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:49.704 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-02T12:00:49.705 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:49.717 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-02T12:00:49.754 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-02T12:00:50.034 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-02T12:00:50.046 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-03-02T12:00:50.074 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-02T12:00:50.084 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-02T12:00:50.086 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-02T12:00:50.124 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-02T12:00:50.379 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-routes. 2021-03-02T12:00:50.390 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-03-02T12:00:50.429 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-02T12:00:50.460 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-rsa. 2021-03-02T12:00:50.472 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-02T12:00:50.510 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-02T12:00:50.706 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-02T12:00:50.717 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:50.756 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-dashboard (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:50.838 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-02T12:00:50.850 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-02T12:00:50.888 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-02T12:00:51.334 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-02T12:00:51.346 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-02T12:00:51.384 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-02T12:00:51.627 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-03-02T12:00:51.640 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:51.668 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:51.729 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-setuptools. 2021-03-02T12:00:51.743 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-02T12:00:51.787 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-02T12:00:52.021 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-numpy. 2021-03-02T12:00:52.034 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-02T12:00:52.071 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-02T12:00:52.174 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-websocket. 2021-03-02T12:00:52.187 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-02T12:00:52.231 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-02T12:00:52.543 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-02T12:00:52.557 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-02T12:00:52.593 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-02T12:00:52.669 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-decorator. 2021-03-02T12:00:52.681 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-02T12:00:52.711 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-02T12:00:53.015 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-scipy. 2021-03-02T12:00:53.113 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-02T12:00:53.113 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-02T12:00:53.114 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-02T12:00:53.114 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:53.115 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-k8sevents (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:53.424 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-02T12:00:53.437 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-02T12:00:53.474 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-02T12:00:53.778 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-02T12:00:53.813 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-02T12:00:53.831 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-02T12:00:54.139 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-02T12:00:54.153 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:54.197 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-mgr-rook (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:54.239 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-02T12:00:54.253 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-02T12:00:54.323 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-02T12:00:54.458 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-02T12:00:54.472 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-02T12:00:54.585 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-02T12:00:54.922 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-02T12:00:55.035 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-02T12:00:55.035 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-02T12:00:55.035 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-joblib. 2021-03-02T12:00:55.035 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-02T12:00:55.036 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-02T12:00:55.216 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package jq. 2021-03-02T12:00:55.429 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-02T12:00:55.430 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-02T12:00:55.430 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-sklearn. 2021-03-02T12:00:55.431 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-02T12:00:55.431 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-02T12:00:55.527 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package socat. 2021-03-02T12:00:55.540 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-02T12:00:55.577 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-02T12:00:55.930 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-02T12:00:55.968 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:00:55.969 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-02T12:00:55.970 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-02T12:00:55.980 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:55.988 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-02T12:00:56.258 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package xmlstarlet. 2021-03-02T12:00:56.558 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-02T12:00:56.559 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-02T12:00:56.654 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package ceph-test. 2021-03-02T12:00:56.858 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../086-ceph-test_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:56.858 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking ceph-test (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:00:57.108 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-monotonic. 2021-03-02T12:00:57.120 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-02T12:00:57.158 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-02T12:00:57.463 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-fasteners. 2021-03-02T12:00:57.473 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-02T12:00:57.521 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-02T12:00:57.805 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-httplib2. 2021-03-02T12:00:57.818 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-02T12:00:57.856 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-02T12:00:58.158 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-02T12:00:58.171 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-02T12:00:58.208 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-02T12:00:58.545 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-02T12:00:58.557 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-02T12:00:58.595 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-02T12:00:58.915 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-rsa. 2021-03-02T12:00:58.927 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-02T12:00:58.964 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-02T12:00:59.285 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-02T12:00:59.296 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-02T12:00:59.334 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-02T12:00:59.789 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-02T12:00:59.802 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-02T12:00:59.839 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-02T12:00:59.957 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-02T12:00:59.969 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:00:59.998 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking libcephfs-dev (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:00.159 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-setuptools. 2021-03-02T12:01:00.171 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-02T12:01:00.209 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-02T12:01:00.276 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package nvme-cli. 2021-03-02T12:01:00.289 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-02T12:01:00.326 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-02T12:01:00.537 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-websocket. 2021-03-02T12:01:00.550 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-02T12:01:00.587 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-02T12:01:00.673 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-attr. 2021-03-02T12:01:00.687 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-02T12:01:00.764 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-02T12:01:00.890 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-02T12:01:00.904 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-02T12:01:00.948 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-02T12:01:01.075 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-funcsigs. 2021-03-02T12:01:01.090 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-02T12:01:01.134 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-02T12:01:01.361 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-02T12:01:01.375 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:01:01.428 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-k8sevents (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:01.428 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-pluggy. 2021-03-02T12:01:01.439 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-02T12:01:01.478 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-02T12:01:01.722 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-02T12:01:01.723 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-py. 2021-03-02T12:01:01.736 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-02T12:01:01.737 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-02T12:01:01.772 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-02T12:01:01.773 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-02T12:01:02.117 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python-pytest. 2021-03-02T12:01:02.129 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-02T12:01:02.142 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-02T12:01:02.155 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-02T12:01:02.167 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-02T12:01:02.195 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-02T12:01:02.454 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-attr. 2021-03-02T12:01:02.465 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-02T12:01:02.487 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-02T12:01:02.495 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-02T12:01:02.501 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.9-47-g3664024e-1bionic_all.deb ... 2021-03-02T12:01:02.545 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-mgr-rook (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:02.798 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-cycler. 2021-03-02T12:01:02.810 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-02T12:01:02.848 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-02T12:01:02.856 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-02T12:01:02.870 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-02T12:01:03.000 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-02T12:01:03.151 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-webencodings. 2021-03-02T12:01:03.247 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-02T12:01:03.247 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-02T12:01:03.395 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-02T12:01:03.407 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-02T12:01:03.445 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-02T12:01:03.513 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-html5lib. 2021-03-02T12:01:03.579 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-02T12:01:03.579 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-02T12:01:03.714 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package jq. 2021-03-02T12:01:03.728 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-02T12:01:03.772 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-02T12:01:03.865 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-02T12:01:03.877 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-02T12:01:03.915 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-02T12:01:04.034 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package socat. 2021-03-02T12:01:04.046 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-02T12:01:04.084 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-02T12:01:04.351 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-02T12:01:04.365 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-02T12:01:04.401 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-02T12:01:04.437 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-02T12:01:04.637 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-02T12:01:04.637 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-02T12:01:04.746 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tz. 2021-03-02T12:01:04.758 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-02T12:01:04.796 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-02T12:01:04.815 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package xmlstarlet. 2021-03-02T12:01:04.829 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-02T12:01:04.873 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-02T12:01:05.090 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-02T12:01:05.102 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-02T12:01:05.140 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-02T12:01:05.228 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package ceph-test. 2021-03-02T12:01:05.240 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../086-ceph-test_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:01:05.278 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking ceph-test (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:05.763 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-olefile. 2021-03-02T12:01:05.779 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-02T12:01:05.812 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-02T12:01:06.140 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pastescript. 2021-03-02T12:01:06.154 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-02T12:01:06.191 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-02T12:01:06.569 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-02T12:01:06.586 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-02T12:01:06.618 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-02T12:01:07.113 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pluggy. 2021-03-02T12:01:07.127 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-02T12:01:07.171 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-02T12:01:07.499 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-py. 2021-03-02T12:01:07.511 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-02T12:01:07.550 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-02T12:01:07.877 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-02T12:01:07.890 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-02T12:01:07.927 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-02T12:01:08.180 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-pytest. 2021-03-02T12:01:08.193 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-02T12:01:08.221 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-02T12:01:08.533 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-simplejson. 2021-03-02T12:01:08.546 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-02T12:01:08.575 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-02T12:01:08.582 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-02T12:01:08.596 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:01:08.641 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking libcephfs-dev (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:08.884 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package nvme-cli. 2021-03-02T12:01:08.899 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-02T12:01:08.903 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-02T12:01:08.916 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-02T12:01:08.934 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-02T12:01:08.953 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-02T12:01:09.289 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python-attr. 2021-03-02T12:01:09.302 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-02T12:01:09.339 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-02T12:01:09.407 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package radosgw. 2021-03-02T12:01:09.420 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../111-radosgw_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:01:09.456 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking radosgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:09.659 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python-funcsigs. 2021-03-02T12:01:09.673 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-02T12:01:09.718 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-02T12:01:09.987 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python-pluggy. 2021-03-02T12:01:10.001 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-02T12:01:10.045 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-02T12:01:10.348 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python-py. 2021-03-02T12:01:10.360 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-02T12:01:10.398 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-02T12:01:10.518 INFO:teuthology.orchestra.run.smithi012.stdout:Selecting previously unselected package rbd-fuse. 2021-03-02T12:01:10.530 INFO:teuthology.orchestra.run.smithi012.stdout:Preparing to unpack .../112-rbd-fuse_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:01:10.593 INFO:teuthology.orchestra.run.smithi012.stdout:Unpacking rbd-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:10.718 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python-pytest. 2021-03-02T12:01:10.731 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-02T12:01:10.768 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-02T12:01:10.868 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-02T12:01:11.113 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-attr. 2021-03-02T12:01:11.125 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-02T12:01:11.163 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-02T12:01:11.164 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-02T12:01:11.466 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-cycler. 2021-03-02T12:01:11.479 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-02T12:01:11.516 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-02T12:01:11.566 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-02T12:01:11.692 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-02T12:01:11.786 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-webencodings. 2021-03-02T12:01:11.801 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-02T12:01:11.852 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-02T12:01:12.139 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-html5lib. 2021-03-02T12:01:12.153 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-02T12:01:12.213 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-02T12:01:12.319 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-02T12:01:12.550 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-02T12:01:12.563 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-02T12:01:12.599 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-02T12:01:12.704 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-02T12:01:12.997 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-02T12:01:13.477 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-02T12:01:13.477 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-02T12:01:13.478 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-02T12:01:13.478 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-02T12:01:13.478 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-tz. 2021-03-02T12:01:13.479 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-02T12:01:13.479 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-02T12:01:13.499 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-02T12:01:13.625 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-02T12:01:13.751 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-02T12:01:14.118 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-02T12:01:14.119 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-02T12:01:14.119 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-02T12:01:14.314 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-02T12:01:14.473 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-olefile. 2021-03-02T12:01:14.775 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-02T12:01:14.776 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-02T12:01:14.777 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-02T12:01:14.834 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pastescript. 2021-03-02T12:01:14.847 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-02T12:01:14.876 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-02T12:01:14.922 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-argparse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:15.215 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-02T12:01:15.255 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-02T12:01:15.268 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.4_amd64.deb ... 2021-03-02T12:01:15.305 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-02T12:01:15.533 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-02T12:01:15.633 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pluggy. 2021-03-02T12:01:15.647 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-02T12:01:15.691 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-02T12:01:15.702 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-02T12:01:16.003 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-py. 2021-03-02T12:01:16.015 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-02T12:01:16.053 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-02T12:01:16.456 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-02T12:01:16.468 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-02T12:01:16.497 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-02T12:01:16.653 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-02T12:01:16.809 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-pytest. 2021-03-02T12:01:17.058 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-02T12:01:17.058 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-02T12:01:17.059 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-02T12:01:17.170 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-simplejson. 2021-03-02T12:01:17.467 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-py (1.5.2-1) ... 2021-03-02T12:01:17.468 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-02T12:01:17.468 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-02T12:01:17.498 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-02T12:01:17.598 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-02T12:01:18.019 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-02T12:01:18.020 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-02T12:01:18.020 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-02T12:01:18.021 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-02T12:01:18.111 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package radosgw. 2021-03-02T12:01:18.125 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../111-radosgw_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:01:18.170 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking radosgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:18.218 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-02T12:01:18.327 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-02T12:01:18.504 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-02T12:01:18.805 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-02T12:01:19.110 INFO:teuthology.orchestra.run.smithi036.stdout:Selecting previously unselected package rbd-fuse. 2021-03-02T12:01:19.133 INFO:teuthology.orchestra.run.smithi036.stdout:Preparing to unpack .../112-rbd-fuse_15.2.9-47-g3664024e-1bionic_amd64.deb ... 2021-03-02T12:01:19.140 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-02T12:01:19.156 INFO:teuthology.orchestra.run.smithi036.stdout:Unpacking rbd-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:19.445 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-02T12:01:19.770 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-02T12:01:19.848 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-02T12:01:20.196 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-02T12:01:20.199 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-02T12:01:20.313 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-02T12:01:20.542 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-02T12:01:20.835 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:20.924 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.4) ... 2021-03-02T12:01:21.317 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-02T12:01:21.494 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user cephadm....done 2021-03-02T12:01:21.610 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-02T12:01:21.658 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-02T12:01:21.912 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-02T12:01:21.976 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-02T12:01:22.102 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-02T12:01:22.104 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-02T12:01:22.211 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-02T12:01:22.230 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-02T12:01:22.538 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-02T12:01:22.573 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-02T12:01:22.830 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-02T12:01:22.897 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-02T12:01:22.959 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-02T12:01:23.202 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-02T12:01:23.318 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:23.511 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-ceph-argparse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:23.620 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-02T12:01:23.812 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-02T12:01:23.912 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-02T12:01:24.038 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-02T12:01:24.097 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-02T12:01:24.239 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-02T12:01:24.389 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-02T12:01:24.749 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-02T12:01:24.847 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-02T12:01:24.880 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-02T12:01:25.127 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-02T12:01:25.201 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-02T12:01:25.428 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-02T12:01:25.720 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-02T12:01:25.852 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-02T12:01:26.046 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-02T12:01:26.129 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python-py (1.5.2-1) ... 2021-03-02T12:01:26.331 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-02T12:01:26.355 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-02T12:01:26.532 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-02T12:01:26.708 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-02T12:01:26.796 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-02T12:01:26.950 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-02T12:01:27.049 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-02T12:01:27.201 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-02T12:01:27.202 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-02T12:01:27.319 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-02T12:01:27.394 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-02T12:01:27.487 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-02T12:01:27.613 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-02T12:01:27.694 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-02T12:01:27.864 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-02T12:01:28.045 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-02T12:01:28.045 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-02T12:01:28.296 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-02T12:01:28.409 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-02T12:01:28.534 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-02T12:01:28.673 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-02T12:01:28.827 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-02T12:01:29.007 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-02T12:01:29.120 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-02T12:01:29.300 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:29.415 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-02T12:01:29.706 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-02T12:01:29.927 INFO:teuthology.orchestra.run.smithi036.stdout:Adding system user cephadm....done 2021-03-02T12:01:30.057 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-02T12:01:30.082 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-02T12:01:30.367 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-02T12:01:30.392 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-02T12:01:30.526 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-02T12:01:30.669 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-02T12:01:30.684 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-02T12:01:30.994 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-02T12:01:31.020 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-02T12:01:31.288 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-02T12:01:31.337 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-02T12:01:31.389 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-02T12:01:31.638 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-02T12:01:31.715 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:31.764 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-02T12:01:32.033 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-02T12:01:32.320 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-02T12:01:32.437 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-02T12:01:32.772 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-02T12:01:32.962 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-02T12:01:33.115 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-02T12:01:33.205 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-02T12:01:33.237 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-02T12:01:33.279 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-02T12:01:33.492 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-02T12:01:33.623 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-02T12:01:33.740 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-02T12:01:33.793 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-02T12:01:34.102 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-02T12:01:34.116 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-02T12:01:34.376 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-02T12:01:34.461 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-02T12:01:34.510 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-02T12:01:34.611 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librados2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:34.746 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-02T12:01:34.779 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-02T12:01:34.947 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-02T12:01:35.314 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:35.314 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libcephfs-dev (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:35.315 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rados (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:35.348 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-02T12:01:35.383 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-02T12:01:35.438 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-02T12:01:35.683 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-02T12:01:35.908 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-02T12:01:35.908 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-02T12:01:35.994 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-02T12:01:36.002 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-02T12:01:36.245 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-02T12:01:36.329 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-02T12:01:36.371 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-02T12:01:36.747 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-02T12:01:36.848 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-02T12:01:37.124 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-02T12:01:37.427 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-02T12:01:37.428 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-02T12:01:37.726 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-02T12:01:38.060 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-02T12:01:38.137 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-02T12:01:38.422 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up libradosstriper1 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:38.564 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-02T12:01:38.566 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-cephfs (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:38.738 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-02T12:01:38.824 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librgw2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:38.950 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:39.022 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-02T12:01:39.170 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-02T12:01:39.172 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-02T12:01:39.331 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-02T12:01:39.434 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up librbd1 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:39.561 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-02T12:01:39.651 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-02T12:01:39.681 INFO:teuthology.orchestra.run.smithi012.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-02T12:01:39.683 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-02T12:01:39.981 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-02T12:01:39.984 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-02T12:01:40.110 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-02T12:01:40.633 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-modules-core (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:40.759 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-02T12:01:41.085 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:41.211 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up rbd-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:41.323 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-02T12:01:41.337 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up python3-rbd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:41.471 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:41.633 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-02T12:01:41.938 INFO:teuthology.orchestra.run.smithi012.stdout:Adding group ceph....done 2021-03-02T12:01:41.967 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-02T12:01:42.110 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-02T12:01:42.308 INFO:teuthology.orchestra.run.smithi012.stdout:Adding system user ceph....done 2021-03-02T12:01:42.478 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-02T12:01:42.484 INFO:teuthology.orchestra.run.smithi012.stdout:Setting system user ceph properties....done 2021-03-02T12:01:42.641 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-02T12:01:42.737 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-02T12:01:42.846 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-02T12:01:42.851 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-02T12:01:42.964 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up librados2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:43.065 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-02T12:01:43.131 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up radosgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:43.358 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libcephfs2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:43.379 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-02T12:01:43.380 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-02T12:01:43.483 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libcephfs-dev (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:43.610 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-rados (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:43.735 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-02T12:01:43.767 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-test (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:43.893 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-base (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:44.062 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-02T12:01:44.245 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-02T12:01:44.364 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-02T12:01:44.536 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:44.680 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-02T12:01:44.756 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-02T12:01:44.756 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-02T12:01:45.238 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-diskprediction-local (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:45.381 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-osd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:45.630 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-02T12:01:45.799 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-02T12:01:45.800 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-02T12:01:46.058 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:46.201 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:46.343 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-rook (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:46.464 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-02T12:01:46.663 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mds (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:46.703 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-02T12:01:46.704 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-02T12:01:46.740 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up libradosstriper1 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:46.849 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-cephfs (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:46.987 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mon (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.150 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up librgw2 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.236 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-02T12:01:47.236 INFO:teuthology.orchestra.run.smithi012.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-02T12:01:47.276 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.497 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-k8sevents (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.699 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-02T12:01:47.700 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-02T12:01:47.700 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph-mgr-dashboard (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:Setting up ceph (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.785 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up librbd1 (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:47.842 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-02T12:01:47.945 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-02T12:01:48.044 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-02T12:01:48.161 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-02T12:01:48.162 INFO:teuthology.orchestra.run.smithi036.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-02T12:01:48.390 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-02T12:01:49.059 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-modules-core (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:49.185 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-02T12:01:49.536 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-rgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:49.670 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up rbd-fuse (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:49.796 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up python3-rbd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:49.922 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-common (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:50.373 INFO:teuthology.orchestra.run.smithi036.stdout:Adding group ceph....done 2021-03-02T12:01:50.638 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-02T12:01:50.673 INFO:teuthology.orchestra.run.smithi012.stdout:ureadahead will be reprofiled on next reboot 2021-03-02T12:01:50.722 INFO:teuthology.orchestra.run.smithi012.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-02T12:01:51.051 INFO:teuthology.orchestra.run.smithi036.stdout:Adding system user ceph....done 2021-03-02T12:01:51.157 INFO:teuthology.orchestra.run.smithi012.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-02T12:01:51.160 DEBUG:teuthology.parallel:result is None 2021-03-02T12:01:51.234 INFO:teuthology.orchestra.run.smithi036.stdout:Setting system user ceph properties....done 2021-03-02T12:01:51.387 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-02T12:01:51.644 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-02T12:01:51.955 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up radosgw (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:52.192 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-02T12:01:52.193 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-02T12:01:52.600 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-test (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:52.726 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-base (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:53.061 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-02T12:01:53.336 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:53.557 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-02T12:01:53.558 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-02T12:01:53.820 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-diskprediction-local (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:53.964 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-osd (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:54.409 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-02T12:01:54.409 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-02T12:01:54.658 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-cephadm (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:54.766 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:54.892 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-rook (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:55.035 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mds (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:55.261 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-02T12:01:55.262 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-02T12:01:55.544 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mon (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:55.772 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-02T12:01:55.772 INFO:teuthology.orchestra.run.smithi036.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-02T12:01:56.422 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-k8sevents (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:56.547 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph-mgr-dashboard (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:56.665 INFO:teuthology.orchestra.run.smithi036.stdout:Setting up ceph (15.2.9-47-g3664024e-1bionic) ... 2021-03-02T12:01:56.855 INFO:teuthology.orchestra.run.smithi036.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-02T12:01:56.967 INFO:teuthology.orchestra.run.smithi036.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-02T12:01:59.187 INFO:teuthology.orchestra.run.smithi036.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-02T12:01:59.229 INFO:teuthology.orchestra.run.smithi036.stdout:ureadahead will be reprofiled on next reboot 2021-03-02T12:01:59.287 INFO:teuthology.orchestra.run.smithi036.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-02T12:01:59.740 INFO:teuthology.orchestra.run.smithi036.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-02T12:01:59.742 DEBUG:teuthology.parallel:result is None 2021-03-02T12:01:59.743 DEBUG:teuthology.orchestra.run.smithi012:> sudo lsb_release -is 2021-03-02T12:01:59.802 INFO:teuthology.orchestra.run.smithi012.stdout:Ubuntu 2021-03-02T12:01:59.808 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-02T12:01:59.809 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3664024eebddedeee285ff2f143b16556af4e85d 2021-03-02T12:01:59.998 DEBUG:teuthology.orchestra.run.smithi012:> dpkg-query -W -f '${Version}' ceph 2021-03-02T12:02:00.064 INFO:teuthology.orchestra.run.smithi012.stdout:15.2.9-47-g3664024e-1bionic 2021-03-02T12:02:00.064 INFO:teuthology.packaging:The installed version of ceph is 15.2.9-47-g3664024e-1bionic 2021-03-02T12:02:00.065 INFO:teuthology.task.install:The correct ceph version 15.2.9-47-g3664024e-1bionic is installed. 2021-03-02T12:02:00.066 DEBUG:teuthology.orchestra.run.smithi036:> sudo lsb_release -is 2021-03-02T12:02:00.121 INFO:teuthology.orchestra.run.smithi036.stdout:Ubuntu 2021-03-02T12:02:00.126 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-02T12:02:00.128 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&sha1=3664024eebddedeee285ff2f143b16556af4e85d 2021-03-02T12:02:00.297 DEBUG:teuthology.orchestra.run.smithi036:> dpkg-query -W -f '${Version}' ceph 2021-03-02T12:02:00.482 INFO:teuthology.orchestra.run.smithi036.stdout:15.2.9-47-g3664024e-1bionic 2021-03-02T12:02:00.483 INFO:teuthology.packaging:The installed version of ceph is 15.2.9-47-g3664024e-1bionic 2021-03-02T12:02:00.484 INFO:teuthology.task.install:The correct ceph version 15.2.9-47-g3664024e-1bionic is installed. 2021-03-02T12:02:00.484 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-02T12:02:00.487 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:00.487 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-02T12:02:00.503 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:00.504 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-02T12:02:00.538 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-02T12:02:00.540 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:00.540 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/daemon-helper 2021-03-02T12:02:00.560 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-02T12:02:00.679 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:00.679 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/usr/bin/daemon-helper 2021-03-02T12:02:00.695 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-02T12:02:00.753 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-02T12:02:00.754 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:00.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-02T12:02:00.769 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-02T12:02:00.835 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:00.835 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-02T12:02:00.850 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-02T12:02:00.905 INFO:teuthology.run_tasks:Running task ceph... 2021-03-02T12:02:01.101 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-02T12:02:01.105 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 777 /var/log/ceph 2021-03-02T12:02:01.139 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 777 /var/log/ceph 2021-03-02T12:02:01.180 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-02T12:02:01.181 DEBUG:teuthology.orchestra.run.smithi012:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-02T12:02:01.211 DEBUG:teuthology.orchestra.run.smithi036:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-02T12:02:01.247 INFO:tasks.ceph:Creating extra log directories... 2021-03-02T12:02:01.247 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-02T12:02:01.277 DEBUG:teuthology.orchestra.run.smithi036:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-02T12:02:01.311 INFO:tasks.ceph:Creating ceph cluster c1... 2021-03-02T12:02:01.312 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-02T12:02:01.312 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536', 'branch': 'wip-yuri-testing-2021-03-01-1125-octopus', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5925536', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2021-03-01-1125-octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '3664024eebddedeee285ff2f143b16556af4e85d'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuri-testing-2021-03-01-1125-octopus', 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2021-03-01-1125-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3664024eebddedeee285ff2f143b16556af4e85d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'targets': {'smithi012.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8QYmvqWVKZIAMULP+iGClqAj+x24U+MP7H3fZe4odr07HfPHg/CnVtjfdT2QyGoU5VWBGYl+SEuXm0F9MLifQPqrx89kTr4LayNySLJQrh+hjJta+XAB+16HRRmbzsRs54FclMEAO+Ogu18SUyjovXIFKv3EztPP8YYLZZ/0jwJPQUDmu1dL8y9dVg7DeVO3oVI2TxHwNoqeYdEedr35bO2P5fJZ2u11BU5j2X0gMI4J6IhdUplsGqUf2MSnMtXrod9tIteE5Qh9Kq8uBdbatCDaRYEULY9sk8PahmGsDPxGeg0gU+TKDKvpUXx2vgVkByYO4eOW0k4zylvLEhi9h', 'smithi036.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIsrda0walO1WCtov3w5mYxsSoKKIQyKQdRRtrtL3bxQ0LNiCw/gthawcXiWIIG8Nh27MjSrXMSVKshdoXMQdf74860hF1Wrpc27wEmlEZgdly3AgAnyxTqvHcORqQLj4p7XZbiwk6uKsbXjWaEdrYGYAPt9/D6CIazdKZKRbAs/04UE4xeS/66T6cnuYPGV96mPMpyCN4FdBTvg7nOARpckzH2UkDMtOjFiT6ebPowrJeTt4qawjPfEcsG4ZeWqQPsI/xqcF3EXWoYd1Rmrobof2UL1bZ7KLD7Fl6tA6oyl1330KFYcKF9wmhCFZnqSW/BYmjtuVkDTWDwhafLP3J'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-01_21:06:29', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.25973'} 2021-03-02T12:02:01.313 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-03-02T12:02:01.340 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-03-02T12:02:01.368 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2021-03-02T12:02:01.395 DEBUG:teuthology.orchestra.run.smithi036:> sudo install -d -m0777 -- /var/run/ceph 2021-03-02T12:02:01.438 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:01.438 DEBUG:teuthology.orchestra.run.smithi012:> dd if=/scratch_devs of=/dev/stdout 2021-03-02T12:02:01.468 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-02T12:02:01.468 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-02T12:02:01.524 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-02T12:02:01.524 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:01.525 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-02T12:02:01.525 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:01.526 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2021-03-02 11:59:21.514311766 +0000 2021-03-02T12:02:01.526 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2021-03-02 11:59:21.390373766 +0000 2021-03-02T12:02:01.526 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2021-03-02 11:59:21.390373766 +0000 2021-03-02T12:02:01.527 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2021-03-02T12:02:01.534 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2021-03-02T12:02:01.534 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2021-03-02T12:02:01.535 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000332244 s, 1.5 MB/s 2021-03-02T12:02:01.538 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-02T12:02:01.597 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-02T12:02:01.598 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:01.598 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-03-02T12:02:01.598 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:01.599 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2021-03-02 11:59:21.750193766 +0000 2021-03-02T12:02:01.599 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2021-03-02 11:59:21.650243767 +0000 2021-03-02T12:02:01.599 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2021-03-02 11:59:21.650243767 +0000 2021-03-02T12:02:01.599 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2021-03-02T12:02:01.607 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2021-03-02T12:02:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2021-03-02T12:02:01.608 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.00038159 s, 1.3 MB/s 2021-03-02T12:02:01.613 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-02T12:02:01.667 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-02T12:02:01.667 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:01.668 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-03-02T12:02:01.668 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:01.668 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2021-03-02 11:59:21.982077767 +0000 2021-03-02T12:02:01.668 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2021-03-02 11:59:21.862137766 +0000 2021-03-02T12:02:01.669 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2021-03-02 11:59:21.862137766 +0000 2021-03-02T12:02:01.669 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2021-03-02T12:02:01.678 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2021-03-02T12:02:01.678 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2021-03-02T12:02:01.679 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000395227 s, 1.3 MB/s 2021-03-02T12:02:01.683 DEBUG:teuthology.orchestra.run.smithi012:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-02T12:02:01.737 INFO:teuthology.orchestra.run.smithi012.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-02T12:02:01.737 INFO:teuthology.orchestra.run.smithi012.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:01.737 INFO:teuthology.orchestra.run.smithi012.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-03-02T12:02:01.738 INFO:teuthology.orchestra.run.smithi012.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:01.738 INFO:teuthology.orchestra.run.smithi012.stdout:Access: 2021-03-02 11:59:22.221957767 +0000 2021-03-02T12:02:01.738 INFO:teuthology.orchestra.run.smithi012.stdout:Modify: 2021-03-02 11:59:22.102017766 +0000 2021-03-02T12:02:01.738 INFO:teuthology.orchestra.run.smithi012.stdout:Change: 2021-03-02 11:59:22.102017766 +0000 2021-03-02T12:02:01.739 INFO:teuthology.orchestra.run.smithi012.stdout: Birth: - 2021-03-02T12:02:01.746 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records in 2021-03-02T12:02:01.747 INFO:teuthology.orchestra.run.smithi012.stderr:1+0 records out 2021-03-02T12:02:01.747 INFO:teuthology.orchestra.run.smithi012.stderr:512 bytes copied, 0.000404584 s, 1.3 MB/s 2021-03-02T12:02:01.751 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:01.752 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-02T12:02:01.752 INFO:tasks.ceph:Generating config... 2021-03-02T12:02:01.755 INFO:tasks.ceph:[client] debug rgw = 20 2021-03-02T12:02:01.755 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-03-02T12:02:01.756 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-03-02T12:02:01.756 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-02T12:02:01.756 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-03-02T12:02:01.757 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-03-02T12:02:01.757 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-03-02T12:02:01.757 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-03-02T12:02:01.757 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-03-02T12:02:01.758 INFO:tasks.ceph:[client] setgroup = ceph 2021-03-02T12:02:01.758 INFO:tasks.ceph:[client] setuser = ceph 2021-03-02T12:02:01.758 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-03-02T12:02:01.759 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-02T12:02:01.759 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-02T12:02:01.759 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-02T12:02:01.760 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-02T12:02:01.760 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-02T12:02:01.760 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-03-02T12:02:01.760 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-02T12:02:01.761 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-02T12:02:01.761 INFO:tasks.ceph:[osd] debug ms = 1 2021-03-02T12:02:01.761 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-02T12:02:01.762 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-03-02T12:02:01.762 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-03-02T12:02:01.762 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-03-02T12:02:01.763 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-03-02T12:02:01.763 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-03-02T12:02:01.763 INFO:tasks.ceph:Setting up c1.mon.a... 2021-03-02T12:02:01.764 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-03-02T12:02:01.840 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/c1.keyring 2021-03-02T12:02:01.842 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-03-02T12:02:01.927 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-02T12:02:02.002 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.12')] 2021-03-02T12:02:02.003 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.12 --print /home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:02.077 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:02.078 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: generated fsid b30fed7c-b977-4b5b-b239-3ed3b12d3d97 2021-03-02T12:02:02.078 INFO:teuthology.orchestra.run.smithi012.stdout:epoch 0 2021-03-02T12:02:02.078 INFO:teuthology.orchestra.run.smithi012.stdout:fsid b30fed7c-b977-4b5b-b239-3ed3b12d3d97 2021-03-02T12:02:02.079 INFO:teuthology.orchestra.run.smithi012.stdout:last_changed 2021-03-02T12:02:02.079659+0000 2021-03-02T12:02:02.079 INFO:teuthology.orchestra.run.smithi012.stdout:created 2021-03-02T12:02:02.079659+0000 2021-03-02T12:02:02.079 INFO:teuthology.orchestra.run.smithi012.stdout:min_mon_release 0 (unknown) 2021-03-02T12:02:02.079 INFO:teuthology.orchestra.run.smithi012.stdout:0: [v2:172.21.15.12:3300/0,v1:172.21.15.12:6789/0] mon.a 2021-03-02T12:02:02.080 INFO:teuthology.orchestra.run.smithi012.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-03-02T12:02:02.081 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID b30fed7c-b977-4b5b-b239-3ed3b12d3d97... 2021-03-02T12:02:02.082 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-03-02T12:02:02.136 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-03-02T12:02:02.166 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2021-03-02T12:02:02.167 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2021-03-02T12:02:02.167 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-02T12:02:02.168 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2021-03-02T12:02:02.168 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.169 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2021-03-02T12:02:02.169 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.169 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2021-03-02T12:02:02.170 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.170 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2021-03-02T12:02:02.171 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2021-03-02T12:02:02.171 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.172 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2021-03-02T12:02:02.172 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2021-03-02T12:02:02.172 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.173 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-02T12:02:02.173 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2021-03-02T12:02:02.174 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.174 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2021-03-02T12:02:02.175 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-02T12:02:02.175 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.176 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2021-03-02T12:02:02.176 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.177 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2021-03-02T12:02:02.177 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2021-03-02T12:02:02.178 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2021-03-02T12:02:02.178 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2021-03-02T12:02:02.178 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2021-03-02T12:02:02.179 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2021-03-02T12:02:02.179 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2021-03-02T12:02:02.180 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-02T12:02:02.180 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-02T12:02:02.181 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.181 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-02T12:02:02.181 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.182 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2021-03-02T12:02:02.182 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2021-03-02T12:02:02.183 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.184 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2021-03-02T12:02:02.184 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2021-03-02T12:02:02.184 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2021-03-02T12:02:02.185 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = 172.21.15.12 2021-03-02T12:02:02.185 INFO:teuthology.orchestra.run.smithi012.stdout: osd heartbeat grace = 40 2021-03-02T12:02:02.185 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = b30fed7c-b977-4b5b-b239-3ed3b12d3d97 2021-03-02T12:02:02.186 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.186 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2021-03-02T12:02:02.187 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2021-03-02T12:02:02.187 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.187 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2021-03-02T12:02:02.187 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2021-03-02T12:02:02.187 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.188 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2021-03-02T12:02:02.188 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2021-03-02T12:02:02.188 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.188 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2021-03-02T12:02:02.189 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2021-03-02T12:02:02.189 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2021-03-02T12:02:02.189 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.189 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2021-03-02T12:02:02.189 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2021-03-02T12:02:02.190 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.190 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2021-03-02T12:02:02.190 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.191 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2021-03-02T12:02:02.191 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.192 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2021-03-02T12:02:02.192 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.192 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2021-03-02T12:02:02.193 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2021-03-02T12:02:02.193 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.194 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2021-03-02T12:02:02.194 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2021-03-02T12:02:02.195 INFO:teuthology.orchestra.run.smithi012.stdout: debug filestore = 20 2021-03-02T12:02:02.196 INFO:teuthology.orchestra.run.smithi012.stdout: debug journal = 20 2021-03-02T12:02:02.197 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2021-03-02T12:02:02.198 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 25 2021-03-02T12:02:02.198 INFO:teuthology.orchestra.run.smithi012.stdout: osd fast shutdown = False 2021-03-02T12:02:02.198 INFO:teuthology.orchestra.run.smithi012.stdout: rocksdb delete range threshold = 0 2021-03-02T12:02:02.199 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.199 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2021-03-02T12:02:02.200 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2021-03-02T12:02:02.200 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2021-03-02T12:02:02.201 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2021-03-02T12:02:02.201 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2021-03-02T12:02:02.201 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:02.202 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:02.202 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2021-03-02T12:02:02.203 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.203 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2021-03-02T12:02:02.205 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2021-03-02T12:02:02.206 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2021-03-02T12:02:02.206 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2021-03-02T12:02:02.207 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2021-03-02T12:02:02.207 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2021-03-02T12:02:02.208 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2021-03-02T12:02:02.208 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:02.208 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2021-03-02T12:02:02.209 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2021-03-02T12:02:02.209 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:02.210 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd crush smoke test = False 2021-03-02T12:02:02.210 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:02.210 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2021-03-02T12:02:02.211 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2021-03-02T12:02:02.211 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2021-03-02T12:02:02.212 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2021-03-02T12:02:02.212 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-02T12:02:02.212 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-02T12:02:02.213 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw = 20 2021-03-02T12:02:02.213 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt require ssl = False 2021-03-02T12:02:02.214 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms backend = testing 2021-03-02T12:02:02.214 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-02T12:02:02.214 INFO:teuthology.orchestra.run.smithi012.stdout: rgw curl low speed time = 300 2021-03-02T12:02:02.215 INFO:teuthology.orchestra.run.smithi012.stdout: rgw data log num shards = 4 2021-03-02T12:02:02.215 INFO:teuthology.orchestra.run.smithi012.stdout: rgw md log max shards = 4 2021-03-02T12:02:02.215 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync log trim interval = 0 2021-03-02T12:02:02.220 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync obj etag verify = True 2021-03-02T12:02:02.220 INFO:teuthology.orchestra.run.smithi012.stdout: setgroup = ceph 2021-03-02T12:02:02.220 INFO:teuthology.orchestra.run.smithi012.stdout: setuser = ceph 2021-03-02T12:02:02.221 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2021-03-02T12:02:02.221 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2021-03-02T12:02:02.222 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10 2021-03-02T12:02:02.222 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2021-03-02T12:02:02.225 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 1000 2021-03-02T12:02:02.225 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2021-03-02T12:02:02.225 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10000 2021-03-02T12:02:02.230 INFO:teuthology.orchestra.run.smithi036.stdout:[global] 2021-03-02T12:02:02.230 INFO:teuthology.orchestra.run.smithi036.stdout: chdir = "" 2021-03-02T12:02:02.231 INFO:teuthology.orchestra.run.smithi036.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-02T12:02:02.231 INFO:teuthology.orchestra.run.smithi036.stdout: auth supported = cephx 2021-03-02T12:02:02.231 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.231 INFO:teuthology.orchestra.run.smithi036.stdout: filestore xattr use omap = true 2021-03-02T12:02:02.232 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.232 INFO:teuthology.orchestra.run.smithi036.stdout: mon clock drift allowed = 1.000 2021-03-02T12:02:02.232 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.232 INFO:teuthology.orchestra.run.smithi036.stdout: osd crush chooseleaf type = 0 2021-03-02T12:02:02.232 INFO:teuthology.orchestra.run.smithi036.stdout: auth debug = true 2021-03-02T12:02:02.233 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.233 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on old message = true 2021-03-02T12:02:02.233 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on bug = true 2021-03-02T12:02:02.233 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.233 INFO:teuthology.orchestra.run.smithi036.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-02T12:02:02.234 INFO:teuthology.orchestra.run.smithi036.stdout: mon pg warn max object skew = 0 2021-03-02T12:02:02.234 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.234 INFO:teuthology.orchestra.run.smithi036.stdout: # disable pg_autoscaler by default for new pools 2021-03-02T12:02:02.234 INFO:teuthology.orchestra.run.smithi036.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-02T12:02:02.235 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.235 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default size = 2 2021-03-02T12:02:02.235 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.235 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow primary affinity = true 2021-03-02T12:02:02.235 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow pg remap = true 2021-03-02T12:02:02.236 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on legacy crush tunables = false 2021-03-02T12:02:02.236 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on crush straw calc version zero = false 2021-03-02T12:02:02.236 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on no sortbitwise = false 2021-03-02T12:02:02.236 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on osd down out interval zero = false 2021-03-02T12:02:02.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on too few osds = false 2021-03-02T12:02:02.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-02T12:02:02.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-02T12:02:02.237 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.237 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-02T12:02:02.238 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.238 INFO:teuthology.orchestra.run.smithi036.stdout: osd default data pool replay window = 5 2021-03-02T12:02:02.238 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.239 INFO:teuthology.orchestra.run.smithi036.stdout: mon allow pool delete = true 2021-03-02T12:02:02.239 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.239 INFO:teuthology.orchestra.run.smithi036.stdout: mon cluster log file level = debug 2021-03-02T12:02:02.239 INFO:teuthology.orchestra.run.smithi036.stdout: debug asserts on shutdown = true 2021-03-02T12:02:02.239 INFO:teuthology.orchestra.run.smithi036.stdout: mon health detail to clog = false 2021-03-02T12:02:02.240 INFO:teuthology.orchestra.run.smithi036.stdout: mon host = 172.21.15.12 2021-03-02T12:02:02.240 INFO:teuthology.orchestra.run.smithi036.stdout: osd heartbeat grace = 40 2021-03-02T12:02:02.240 INFO:teuthology.orchestra.run.smithi036.stdout: fsid = b30fed7c-b977-4b5b-b239-3ed3b12d3d97 2021-03-02T12:02:02.240 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.242 INFO:teuthology.orchestra.run.smithi036.stdout:[osd] 2021-03-02T12:02:02.242 INFO:teuthology.orchestra.run.smithi036.stdout: osd journal size = 100 2021-03-02T12:02:02.242 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.243 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub load threshold = 5.0 2021-03-02T12:02:02.243 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub max interval = 600 2021-03-02T12:02:02.244 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.244 INFO:teuthology.orchestra.run.smithi036.stdout: osd recover clone overlap = true 2021-03-02T12:02:02.244 INFO:teuthology.orchestra.run.smithi036.stdout: osd recovery max chunk = 1048576 2021-03-02T12:02:02.245 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.245 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug shutdown = true 2021-03-02T12:02:02.246 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug op order = true 2021-03-02T12:02:02.246 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug verify stray on activate = true 2021-03-02T12:02:02.247 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.247 INFO:teuthology.orchestra.run.smithi036.stdout: osd open classes on start = true 2021-03-02T12:02:02.247 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug pg log writeout = true 2021-03-02T12:02:02.248 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.248 INFO:teuthology.orchestra.run.smithi036.stdout: osd deep scrub update digest min age = 30 2021-03-02T12:02:02.248 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.249 INFO:teuthology.orchestra.run.smithi036.stdout: osd map max advance = 10 2021-03-02T12:02:02.249 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.250 INFO:teuthology.orchestra.run.smithi036.stdout: journal zero on create = true 2021-03-02T12:02:02.250 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.250 INFO:teuthology.orchestra.run.smithi036.stdout: filestore ondisk finisher threads = 3 2021-03-02T12:02:02.251 INFO:teuthology.orchestra.run.smithi036.stdout: filestore apply finisher threads = 3 2021-03-02T12:02:02.251 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.251 INFO:teuthology.orchestra.run.smithi036.stdout: bdev debug aio = true 2021-03-02T12:02:02.252 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug misdirected ops = true 2021-03-02T12:02:02.252 INFO:teuthology.orchestra.run.smithi036.stdout: debug filestore = 20 2021-03-02T12:02:02.253 INFO:teuthology.orchestra.run.smithi036.stdout: debug journal = 20 2021-03-02T12:02:02.253 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2021-03-02T12:02:02.253 INFO:teuthology.orchestra.run.smithi036.stdout: debug osd = 25 2021-03-02T12:02:02.254 INFO:teuthology.orchestra.run.smithi036.stdout: osd fast shutdown = False 2021-03-02T12:02:02.254 INFO:teuthology.orchestra.run.smithi036.stdout: rocksdb delete range threshold = 0 2021-03-02T12:02:02.255 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.255 INFO:teuthology.orchestra.run.smithi036.stdout:[mgr] 2021-03-02T12:02:02.255 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2021-03-02T12:02:02.256 INFO:teuthology.orchestra.run.smithi036.stdout: debug mgr = 20 2021-03-02T12:02:02.256 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2021-03-02T12:02:02.256 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2021-03-02T12:02:02.257 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:02.257 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:02.258 INFO:teuthology.orchestra.run.smithi036.stdout: mgr/telemetry/nag = false 2021-03-02T12:02:02.258 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.258 INFO:teuthology.orchestra.run.smithi036.stdout:[mon] 2021-03-02T12:02:02.259 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2021-03-02T12:02:02.259 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2021-03-02T12:02:02.259 INFO:teuthology.orchestra.run.smithi036.stdout: debug paxos = 20 2021-03-02T12:02:02.259 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2021-03-02T12:02:02.259 INFO:teuthology.orchestra.run.smithi036.stdout: mon data avail warn = 5 2021-03-02T12:02:02.260 INFO:teuthology.orchestra.run.smithi036.stdout: mon mgr mkfs grace = 240 2021-03-02T12:02:02.260 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:02.260 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd reporter subtree level = osd 2021-03-02T12:02:02.260 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd prime pg temp = true 2021-03-02T12:02:02.261 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:02.261 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd crush smoke test = False 2021-03-02T12:02:02.261 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:02.262 INFO:teuthology.orchestra.run.smithi036.stdout:[client] 2021-03-02T12:02:02.262 INFO:teuthology.orchestra.run.smithi036.stdout: rgw cache enabled = true 2021-03-02T12:02:02.263 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable ops log = true 2021-03-02T12:02:02.263 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable usage log = true 2021-03-02T12:02:02.263 INFO:teuthology.orchestra.run.smithi036.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-02T12:02:02.264 INFO:teuthology.orchestra.run.smithi036.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-02T12:02:02.264 INFO:teuthology.orchestra.run.smithi036.stdout: debug rgw = 20 2021-03-02T12:02:02.265 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt require ssl = False 2021-03-02T12:02:02.265 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms backend = testing 2021-03-02T12:02:02.268 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-02T12:02:02.268 INFO:teuthology.orchestra.run.smithi036.stdout: rgw curl low speed time = 300 2021-03-02T12:02:02.268 INFO:teuthology.orchestra.run.smithi036.stdout: rgw data log num shards = 4 2021-03-02T12:02:02.269 INFO:teuthology.orchestra.run.smithi036.stdout: rgw md log max shards = 4 2021-03-02T12:02:02.269 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync log trim interval = 0 2021-03-02T12:02:02.270 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync obj etag verify = True 2021-03-02T12:02:02.270 INFO:teuthology.orchestra.run.smithi036.stdout: setgroup = ceph 2021-03-02T12:02:02.270 INFO:teuthology.orchestra.run.smithi036.stdout: setuser = ceph 2021-03-02T12:02:02.271 INFO:teuthology.orchestra.run.smithi036.stdout:[mon.a] 2021-03-02T12:02:02.273 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.0] 2021-03-02T12:02:02.273 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10 2021-03-02T12:02:02.273 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.1] 2021-03-02T12:02:02.274 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 1000 2021-03-02T12:02:02.274 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.2] 2021-03-02T12:02:02.275 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10000 2021-03-02T12:02:02.276 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-03-02T12:02:02.276 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-03-02T12:02:02.333 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-02T12:02:02.397 DEBUG:teuthology.orchestra.remote:smithi012:/etc/ceph/c1.keyring is 206B 2021-03-02T12:02:02.473 DEBUG:teuthology.orchestra.remote:smithi012:/home/ubuntu/cephtest/c1.monmap is 179B 2021-03-02T12:02:02.517 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2021-03-02T12:02:02.518 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:02.518 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/c1.keyring 2021-03-02T12:02:02.518 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-02T12:02:02.541 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:02.542 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:02.593 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi036.front.sepia.ceph.com 2021-03-02T12:02:02.594 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:02.594 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/ceph/c1.keyring 2021-03-02T12:02:02.594 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-02T12:02:02.623 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:02.623 DEBUG:teuthology.orchestra.run.smithi036:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:02.675 INFO:tasks.ceph:Setting up mon nodes... 2021-03-02T12:02:02.675 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-02T12:02:02.676 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-03-02T12:02:02.721 INFO:teuthology.orchestra.run.smithi012.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-03-02T12:02:02.723 INFO:tasks.ceph:Setting up mds nodes... 2021-03-02T12:02:02.724 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-02T12:02:02.724 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-03-02T12:02:02.798 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/c1.client.0.keyring 2021-03-02T12:02:02.811 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c1.client.1.keyring && sudo chmod 0644 /etc/ceph/c1.client.1.keyring 2021-03-02T12:02:02.886 INFO:teuthology.orchestra.run.smithi012.stdout:creating /etc/ceph/c1.client.1.keyring 2021-03-02T12:02:02.898 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-02T12:02:02.898 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-02T12:02:02.899 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-03-02T12:02:02.960 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:02.960 INFO:tasks.ceph:role: c1.osd.0 2021-03-02T12:02:02.961 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com 2021-03-02T12:02:02.962 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-02T12:02:03.473 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-02T12:02:03.473 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-02T12:02:03.473 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-02T12:02:03.474 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-02T12:02:03.474 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2021-03-02T12:02:03.474 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-02T12:02:03.474 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-02T12:02:03.475 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-02T12:02:03.475 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-02T12:02:03.476 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2021-03-02T12:02:03.476 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-03-02T12:02:03.549 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-03-02T12:02:03.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-02T12:02:03.564 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2021-03-02T12:02:03.564 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-03-02T12:02:03.621 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:03.621 INFO:tasks.ceph:role: c1.osd.1 2021-03-02T12:02:03.621 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com 2021-03-02T12:02:03.622 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-02T12:02:04.130 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-02T12:02:04.131 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-02T12:02:04.131 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-02T12:02:04.131 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-02T12:02:04.132 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2021-03-02T12:02:04.132 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-02T12:02:04.132 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-02T12:02:04.132 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-02T12:02:04.132 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-02T12:02:04.134 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2021-03-02T12:02:04.134 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-03-02T12:02:04.208 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-03-02T12:02:04.221 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2021-03-02T12:02:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-02T12:02:04.222 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-03-02T12:02:04.281 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:04.281 INFO:tasks.ceph:role: c1.osd.2 2021-03-02T12:02:04.282 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com 2021-03-02T12:02:04.282 DEBUG:teuthology.orchestra.run.smithi012:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-02T12:02:04.784 INFO:teuthology.orchestra.run.smithi012.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-02T12:02:04.785 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-02T12:02:04.785 INFO:teuthology.orchestra.run.smithi012.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-02T12:02:04.785 INFO:teuthology.orchestra.run.smithi012.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-02T12:02:04.786 INFO:teuthology.orchestra.run.smithi012.stdout: = sunit=0 swidth=0 blks 2021-03-02T12:02:04.786 INFO:teuthology.orchestra.run.smithi012.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-02T12:02:04.786 INFO:teuthology.orchestra.run.smithi012.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-02T12:02:04.786 INFO:teuthology.orchestra.run.smithi012.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-02T12:02:04.787 INFO:teuthology.orchestra.run.smithi012.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-02T12:02:04.788 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi012.front.sepia.ceph.com -o noatime 2021-03-02T12:02:04.789 DEBUG:teuthology.orchestra.run.smithi012:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-03-02T12:02:04.841 DEBUG:teuthology.orchestra.run.smithi012:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-03-02T12:02:04.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-02T12:02:04.894 INFO:teuthology.orchestra.run.smithi012.stderr:sudo: /sbin/restorecon: command not found 2021-03-02T12:02:04.894 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:04.995 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:04.994+0000 7fc3b1dc0d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-03-02T12:02:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:04.994+0000 7fc3b1dc0d80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-03-02T12:02:04.996 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:04.994+0000 7fc3b1dc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-02T12:02:04.997 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:04.994+0000 7fc3b1dc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-02T12:02:04.997 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:04.994+0000 7fc3b1dc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-02T12:02:04.997 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:04.998+0000 7fc3b1dc0d80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-03-02T12:02:05.014 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:05.014+0000 7fc3b1dc0d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-02T12:02:06.910 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-03-02T12:02:06.925 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:07.009 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.006+0000 7fb4866e2d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-03-02T12:02:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.006+0000 7fb4866e2d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-03-02T12:02:07.010 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.006+0000 7fb4866e2d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-02T12:02:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.006+0000 7fb4866e2d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-02T12:02:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.006+0000 7fb4866e2d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-02T12:02:07.011 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.006+0000 7fb4866e2d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-03-02T12:02:07.025 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:07.022+0000 7fb4866e2d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-02T12:02:08.917 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-03-02T12:02:08.977 DEBUG:teuthology.orchestra.run.smithi012:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.058+0000 7f5853febd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-03-02T12:02:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.058+0000 7f5853febd80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-03-02T12:02:09.060 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.058+0000 7f5853febd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-02T12:02:09.061 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.058+0000 7f5853febd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-02T12:02:09.061 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.058+0000 7f5853febd80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-02T12:02:09.061 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.058+0000 7f5853febd80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-03-02T12:02:09.077 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:02:09.074+0000 7f5853febd80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-02T12:02:10.976 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-03-02T12:02:11.039 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-02T12:02:11.039 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:11.087 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.jN16Rx6gyy 2021-03-02T12:02:11.088 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/mgr/c1-x/keyring /tmp/tmp.jN16Rx6gyy 2021-03-02T12:02:11.144 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.jN16Rx6gyy 2021-03-02T12:02:11.270 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.jN16Rx6gyy is 56B 2021-03-02T12:02:11.334 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.jN16Rx6gyy 2021-03-02T12:02:11.343 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:11.387 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.OIQXuJxIpT 2021-03-02T12:02:11.387 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/osd/c1-0/keyring /tmp/tmp.OIQXuJxIpT 2021-03-02T12:02:11.445 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.OIQXuJxIpT 2021-03-02T12:02:11.554 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.OIQXuJxIpT is 56B 2021-03-02T12:02:11.565 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.OIQXuJxIpT 2021-03-02T12:02:11.574 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:11.630 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.SCySeN5q9r 2021-03-02T12:02:11.631 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/osd/c1-1/keyring /tmp/tmp.SCySeN5q9r 2021-03-02T12:02:11.713 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.SCySeN5q9r 2021-03-02T12:02:11.916 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.SCySeN5q9r is 56B 2021-03-02T12:02:12.011 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.SCySeN5q9r 2021-03-02T12:02:12.031 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:12.056 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.nF5KJmuzVt 2021-03-02T12:02:12.057 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/osd/c1-2/keyring /tmp/tmp.nF5KJmuzVt 2021-03-02T12:02:12.124 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.nF5KJmuzVt 2021-03-02T12:02:12.241 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.nF5KJmuzVt is 56B 2021-03-02T12:02:12.260 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.nF5KJmuzVt 2021-03-02T12:02:12.329 DEBUG:teuthology.orchestra.remote:smithi012:/etc/ceph/c1.client.0.keyring is 59B 2021-03-02T12:02:12.431 DEBUG:teuthology.orchestra.remote:smithi012:/etc/ceph/c1.client.1.keyring is 59B 2021-03-02T12:02:12.455 INFO:tasks.ceph:Adding keys to all mons... 2021-03-02T12:02:12.456 DEBUG:teuthology.orchestra.run.smithi012:> sudo tee -a /etc/ceph/c1.keyring 2021-03-02T12:02:12.473 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr.x] 2021-03-02T12:02:12.473 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA6KT5gXrQZKxAAKIogDWxWj4wABVWo0vu0lg== 2021-03-02T12:02:12.473 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2021-03-02T12:02:12.474 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA8KT5gdiRwOxAA1D3OQsS7pV8n2AhD2RjlKg== 2021-03-02T12:02:12.474 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2021-03-02T12:02:12.474 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA/KT5gWv6XABAAxxaTjk5BBuorIna91CRVfA== 2021-03-02T12:02:12.474 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2021-03-02T12:02:12.475 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQBBKT5gTRKpAxAAy3SlXyHWvDinHCRMb4xHpw== 2021-03-02T12:02:12.475 INFO:teuthology.orchestra.run.smithi012.stdout:[client.0] 2021-03-02T12:02:12.475 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA6KT5gpYG2LxAA3FzX8JgdUbHhD0b2XtVJdw== 2021-03-02T12:02:12.475 INFO:teuthology.orchestra.run.smithi012.stdout:[client.1] 2021-03-02T12:02:12.475 INFO:teuthology.orchestra.run.smithi012.stdout: key = AQA6KT5gP1DvNBAAbqTfD+ccPkO6HK8VOp2tBg== 2021-03-02T12:02:12.477 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-02T12:02:12.566 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-02T12:02:12.660 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-02T12:02:12.754 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-02T12:02:12.844 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-02T12:02:12.941 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-02T12:02:13.034 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-02T12:02:13.035 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-03-02T12:02:13.099 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-03-02T12:02:13.211 DEBUG:teuthology.orchestra.run.smithi012:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-03-02T12:02:13.234 DEBUG:teuthology.orchestra.run.smithi012:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-03-02T12:02:13.312 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-03-02T12:02:13.313 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-03-02T12:02:13.317 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-03-02T12:02:13.367 INFO:tasks.ceph.c1.mon.a:Started 2021-03-02T12:02:13.370 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-03-02T12:02:13.371 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-03-02T12:02:13.372 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-03-02T12:02:13.381 INFO:tasks.ceph.c1.mgr.x:Started 2021-03-02T12:02:13.382 INFO:tasks.ceph:Setting crush tunables to default 2021-03-02T12:02:13.382 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd crush tunables default 2021-03-02T12:02:13.735 INFO:teuthology.orchestra.run.smithi012.stderr:adjusted tunables profile to default 2021-03-02T12:02:13.742 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-03-02T12:02:13.742 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:13.749 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.rWOfdnxKbB 2021-03-02T12:02:13.750 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/osd/c1-0/fsid /tmp/tmp.rWOfdnxKbB 2021-03-02T12:02:13.807 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.rWOfdnxKbB 2021-03-02T12:02:13.913 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.rWOfdnxKbB is 37B 2021-03-02T12:02:13.925 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.rWOfdnxKbB 2021-03-02T12:02:13.935 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:13.983 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.VcbsUroFt7 2021-03-02T12:02:13.984 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/osd/c1-1/fsid /tmp/tmp.VcbsUroFt7 2021-03-02T12:02:14.040 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.VcbsUroFt7 2021-03-02T12:02:14.146 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.VcbsUroFt7 is 37B 2021-03-02T12:02:14.157 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.VcbsUroFt7 2021-03-02T12:02:14.163 DEBUG:teuthology.orchestra.run.smithi012:> mktemp 2021-03-02T12:02:14.211 INFO:teuthology.orchestra.run.smithi012.stdout:/tmp/tmp.cC2ScYKXpE 2021-03-02T12:02:14.212 DEBUG:teuthology.orchestra.run.smithi012:> sudo cp /var/lib/ceph/osd/c1-2/fsid /tmp/tmp.cC2ScYKXpE 2021-03-02T12:02:14.268 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0666 /tmp/tmp.cC2ScYKXpE 2021-03-02T12:02:14.373 DEBUG:teuthology.orchestra.remote:smithi012:/tmp/tmp.cC2ScYKXpE is 37B 2021-03-02T12:02:14.430 DEBUG:teuthology.orchestra.run.smithi012:> rm -fr /tmp/tmp.cC2ScYKXpE 2021-03-02T12:02:14.437 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd new a5e5365c-9b67-463e-a4c8-8256b20bee9c 0 2021-03-02T12:02:14.688 INFO:teuthology.orchestra.run.smithi012.stdout:0 2021-03-02T12:02:14.700 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd new 7e09402b-b85a-4cf3-96e4-39eb763b35c5 1 2021-03-02T12:02:14.902 INFO:teuthology.orchestra.run.smithi012.stdout:1 2021-03-02T12:02:14.919 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd new ac84c139-7b85-4f4d-ac03-5e0cd413d1d0 2 2021-03-02T12:02:15.120 INFO:teuthology.orchestra.run.smithi012.stdout:2 2021-03-02T12:02:15.133 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-03-02T12:02:15.134 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-03-02T12:02:15.136 INFO:tasks.ceph.c1.osd.0:Started 2021-03-02T12:02:15.137 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-03-02T12:02:15.137 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-03-02T12:02:15.140 INFO:tasks.ceph.c1.osd.1:Started 2021-03-02T12:02:15.140 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-03-02T12:02:15.141 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-03-02T12:02:15.143 INFO:tasks.ceph.c1.osd.2:Started 2021-03-02T12:02:15.144 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-02T12:02:15.446 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-03-02T12:02:15.630 INFO:teuthology.misc.health.smithi012.stdout: 2021-03-02T12:02:15.630 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":5,"fsid":"b30fed7c-b977-4b5b-b239-3ed3b12d3d97","created":"2021-03-02T12:02:13.499041+0000","modified":"2021-03-02T12:02:15.119364+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"a5e5365c-9b67-463e-a4c8-8256b20bee9c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7e09402b-b85a-4cf3-96e4-39eb763b35c5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ac84c139-7b85-4f4d-ac03-5e0cd413d1d0","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:15.641 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-03-02T12:02:15.740 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2021-03-02T12:02:15.738+0000 7f14ef427d80 -1 Falling back to public interface 2021-03-02T12:02:15.741 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2021-03-02T12:02:15.738+0000 7f81df4e4d80 -1 Falling back to public interface 2021-03-02T12:02:15.755 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2021-03-02T12:02:15.754+0000 7f76e80fdd80 -1 Falling back to public interface 2021-03-02T12:02:16.116 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2021-03-02T12:02:16.114+0000 7f81df4e4d80 -1 osd.1 0 log_to_monitors {default=true} 2021-03-02T12:02:16.340 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2021-03-02T12:02:16.338+0000 7f14ef427d80 -1 osd.0 0 log_to_monitors {default=true} 2021-03-02T12:02:16.361 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2021-03-02T12:02:16.358+0000 7f76e80fdd80 -1 osd.2 0 log_to_monitors {default=true} 2021-03-02T12:02:17.531 INFO:tasks.ceph.c1.osd.1.smithi012.stderr:2021-03-02T12:02:17.530+0000 7f81c8f30700 -1 osd.1 0 waiting for initial osdmap 2021-03-02T12:02:17.531 INFO:tasks.ceph.c1.osd.0.smithi012.stderr:2021-03-02T12:02:17.530+0000 7f14d8e73700 -1 osd.0 0 waiting for initial osdmap 2021-03-02T12:02:17.532 INFO:tasks.ceph.c1.osd.2.smithi012.stderr:2021-03-02T12:02:17.530+0000 7f76d1b49700 -1 osd.2 0 waiting for initial osdmap 2021-03-02T12:02:21.946 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-03-02T12:02:22.208 INFO:teuthology.misc.health.smithi012.stdout: 2021-03-02T12:02:22.464 INFO:teuthology.misc.health.smithi012.stdout:{"epoch":10,"fsid":"b30fed7c-b977-4b5b-b239-3ed3b12d3d97","created":"2021-03-02T12:02:13.499041+0000","modified":"2021-03-02T12:02:20.552325+0000","last_up_change":"2021-03-02T12:02:18.532888+0000","last_in_change":"2021-03-02T12:02:18.532888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:17.508284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"a5e5365c-9b67-463e-a4c8-8256b20bee9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":11574},{"type":"v1","addr":"172.21.15.12:6801","nonce":11574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":11574},{"type":"v1","addr":"172.21.15.12:6804","nonce":11574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":11574},{"type":"v1","addr":"172.21.15.12:6812","nonce":11574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":11574},{"type":"v1","addr":"172.21.15.12:6808","nonce":11574}]},"public_addr":"172.21.15.12:6801/11574","cluster_addr":"172.21.15.12:6804/11574","heartbeat_back_addr":"172.21.15.12:6812/11574","heartbeat_front_addr":"172.21.15.12:6808/11574","state":["exists","up"]},{"osd":1,"uuid":"7e09402b-b85a-4cf3-96e4-39eb763b35c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":11573},{"type":"v1","addr":"172.21.15.12:6805","nonce":11573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":11573},{"type":"v1","addr":"172.21.15.12:6809","nonce":11573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":11573},{"type":"v1","addr":"172.21.15.12:6815","nonce":11573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":11573},{"type":"v1","addr":"172.21.15.12:6813","nonce":11573}]},"public_addr":"172.21.15.12:6805/11573","cluster_addr":"172.21.15.12:6809/11573","heartbeat_back_addr":"172.21.15.12:6815/11573","heartbeat_front_addr":"172.21.15.12:6813/11573","state":["exists","up"]},{"osd":2,"uuid":"ac84c139-7b85-4f4d-ac03-5e0cd413d1d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":11575},{"type":"v1","addr":"172.21.15.12:6817","nonce":11575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":11575},{"type":"v1","addr":"172.21.15.12:6819","nonce":11575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":11575},{"type":"v1","addr":"172.21.15.12:6823","nonce":11575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":11575},{"type":"v1","addr":"172.21.15.12:6821","nonce":11575}]},"public_addr":"172.21.15.12:6817/11575","cluster_addr":"172.21.15.12:6819/11575","heartbeat_back_addr":"172.21.15.12:6823/11575","heartbeat_front_addr":"172.21.15.12:6821/11575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.334490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.154996+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.391866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:22.466 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-03-02T12:02:22.467 INFO:tasks.ceph:Creating RBD pool 2021-03-02T12:02:22.467 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd pool create rbd 8 2021-03-02T12:02:23.580 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'rbd' created 2021-03-02T12:02:23.593 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-03-02T12:02:24.591 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rbd' on pool 'rbd' 2021-03-02T12:02:24.604 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-03-02T12:02:24.605 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-03-02T12:02:24.606 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-03-02T12:02:24.881 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:24.881 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":12,"fsid":"b30fed7c-b977-4b5b-b239-3ed3b12d3d97","created":"2021-03-02T12:02:13.499041+0000","modified":"2021-03-02T12:02:24.572025+0000","last_up_change":"2021-03-02T12:02:18.532888+0000","last_in_change":"2021-03-02T12:02:18.532888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:17.508284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-02T12:02:22.751484+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a5e5365c-9b67-463e-a4c8-8256b20bee9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":11574},{"type":"v1","addr":"172.21.15.12:6801","nonce":11574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":11574},{"type":"v1","addr":"172.21.15.12:6804","nonce":11574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":11574},{"type":"v1","addr":"172.21.15.12:6812","nonce":11574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":11574},{"type":"v1","addr":"172.21.15.12:6808","nonce":11574}]},"public_addr":"172.21.15.12:6801/11574","cluster_addr":"172.21.15.12:6804/11574","heartbeat_back_addr":"172.21.15.12:6812/11574","heartbeat_front_addr":"172.21.15.12:6808/11574","state":["exists","up"]},{"osd":1,"uuid":"7e09402b-b85a-4cf3-96e4-39eb763b35c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":11573},{"type":"v1","addr":"172.21.15.12:6805","nonce":11573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":11573},{"type":"v1","addr":"172.21.15.12:6809","nonce":11573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":11573},{"type":"v1","addr":"172.21.15.12:6815","nonce":11573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":11573},{"type":"v1","addr":"172.21.15.12:6813","nonce":11573}]},"public_addr":"172.21.15.12:6805/11573","cluster_addr":"172.21.15.12:6809/11573","heartbeat_back_addr":"172.21.15.12:6815/11573","heartbeat_front_addr":"172.21.15.12:6813/11573","state":["exists","up"]},{"osd":2,"uuid":"ac84c139-7b85-4f4d-ac03-5e0cd413d1d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":11575},{"type":"v1","addr":"172.21.15.12:6817","nonce":11575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":11575},{"type":"v1","addr":"172.21.15.12:6819","nonce":11575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":11575},{"type":"v1","addr":"172.21.15.12:6823","nonce":11575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":11575},{"type":"v1","addr":"172.21.15.12:6821","nonce":11575}]},"public_addr":"172.21.15.12:6817/11575","cluster_addr":"172.21.15.12:6819/11575","heartbeat_back_addr":"172.21.15.12:6823/11575","heartbeat_front_addr":"172.21.15.12:6821/11575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.334490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.154996+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.391866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:24.894 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-02T12:02:17.508284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-03-02T12:02:22.751484+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-03-02T12:02:24.895 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get device_health_metrics pg_num 2021-03-02T12:02:25.177 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 1 2021-03-02T12:02:25.190 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd pool get rbd pg_num 2021-03-02T12:02:25.466 INFO:teuthology.orchestra.run.smithi012.stdout:pg_num: 8 2021-03-02T12:02:25.484 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-03-02T12:02:25.485 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-03-02T12:02:25.485 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early mgr dump --format=json 2021-03-02T12:02:25.798 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:25.810 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6824","nonce":11368},{"type":"v1","addr":"172.21.15.12:6825","nonce":11368}]},"active_addr":"172.21.15.12:6825/11368","active_change":"2021-03-02T12:02:17.457643+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":3210736895}]},{"addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":2610115923}]},{"addrvec":[{"type":"v2","addr":"172.21.15.12:0","nonce":406643683}]}]}} 2021-03-02T12:02:25.813 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-03-02T12:02:25.813 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-03-02T12:02:25.816 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-03-02T12:02:26.090 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:26.091 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"fsid":"b30fed7c-b977-4b5b-b239-3ed3b12d3d97","created":"2021-03-02T12:02:13.499041+0000","modified":"2021-03-02T12:02:25.578429+0000","last_up_change":"2021-03-02T12:02:18.532888+0000","last_in_change":"2021-03-02T12:02:18.532888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:17.508284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-02T12:02:22.751484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a5e5365c-9b67-463e-a4c8-8256b20bee9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":11574},{"type":"v1","addr":"172.21.15.12:6801","nonce":11574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":11574},{"type":"v1","addr":"172.21.15.12:6804","nonce":11574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":11574},{"type":"v1","addr":"172.21.15.12:6812","nonce":11574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":11574},{"type":"v1","addr":"172.21.15.12:6808","nonce":11574}]},"public_addr":"172.21.15.12:6801/11574","cluster_addr":"172.21.15.12:6804/11574","heartbeat_back_addr":"172.21.15.12:6812/11574","heartbeat_front_addr":"172.21.15.12:6808/11574","state":["exists","up"]},{"osd":1,"uuid":"7e09402b-b85a-4cf3-96e4-39eb763b35c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":11573},{"type":"v1","addr":"172.21.15.12:6805","nonce":11573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":11573},{"type":"v1","addr":"172.21.15.12:6809","nonce":11573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":11573},{"type":"v1","addr":"172.21.15.12:6815","nonce":11573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":11573},{"type":"v1","addr":"172.21.15.12:6813","nonce":11573}]},"public_addr":"172.21.15.12:6805/11573","cluster_addr":"172.21.15.12:6809/11573","heartbeat_back_addr":"172.21.15.12:6815/11573","heartbeat_front_addr":"172.21.15.12:6813/11573","state":["exists","up"]},{"osd":2,"uuid":"ac84c139-7b85-4f4d-ac03-5e0cd413d1d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":11575},{"type":"v1","addr":"172.21.15.12:6817","nonce":11575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":11575},{"type":"v1","addr":"172.21.15.12:6819","nonce":11575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":11575},{"type":"v1","addr":"172.21.15.12:6823","nonce":11575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":11575},{"type":"v1","addr":"172.21.15.12:6821","nonce":11575}]},"public_addr":"172.21.15.12:6817/11575","cluster_addr":"172.21.15.12:6819/11575","heartbeat_back_addr":"172.21.15.12:6823/11575","heartbeat_front_addr":"172.21.15.12:6821/11575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.334490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.154996+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.391866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:26.110 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-03-02T12:02:26.111 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd dump --format=json 2021-03-02T12:02:26.380 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:26.381 INFO:teuthology.orchestra.run.smithi012.stdout:{"epoch":13,"fsid":"b30fed7c-b977-4b5b-b239-3ed3b12d3d97","created":"2021-03-02T12:02:13.499041+0000","modified":"2021-03-02T12:02:25.578429+0000","last_up_change":"2021-03-02T12:02:18.532888+0000","last_in_change":"2021-03-02T12:02:18.532888+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:17.508284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-02T12:02:22.751484+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a5e5365c-9b67-463e-a4c8-8256b20bee9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6800","nonce":11574},{"type":"v1","addr":"172.21.15.12:6801","nonce":11574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6802","nonce":11574},{"type":"v1","addr":"172.21.15.12:6804","nonce":11574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6810","nonce":11574},{"type":"v1","addr":"172.21.15.12:6812","nonce":11574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6806","nonce":11574},{"type":"v1","addr":"172.21.15.12:6808","nonce":11574}]},"public_addr":"172.21.15.12:6801/11574","cluster_addr":"172.21.15.12:6804/11574","heartbeat_back_addr":"172.21.15.12:6812/11574","heartbeat_front_addr":"172.21.15.12:6808/11574","state":["exists","up"]},{"osd":1,"uuid":"7e09402b-b85a-4cf3-96e4-39eb763b35c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6803","nonce":11573},{"type":"v1","addr":"172.21.15.12:6805","nonce":11573}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6807","nonce":11573},{"type":"v1","addr":"172.21.15.12:6809","nonce":11573}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6814","nonce":11573},{"type":"v1","addr":"172.21.15.12:6815","nonce":11573}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6811","nonce":11573},{"type":"v1","addr":"172.21.15.12:6813","nonce":11573}]},"public_addr":"172.21.15.12:6805/11573","cluster_addr":"172.21.15.12:6809/11573","heartbeat_back_addr":"172.21.15.12:6815/11573","heartbeat_front_addr":"172.21.15.12:6813/11573","state":["exists","up"]},{"osd":2,"uuid":"ac84c139-7b85-4f4d-ac03-5e0cd413d1d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6816","nonce":11575},{"type":"v1","addr":"172.21.15.12:6817","nonce":11575}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6818","nonce":11575},{"type":"v1","addr":"172.21.15.12:6819","nonce":11575}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6822","nonce":11575},{"type":"v1","addr":"172.21.15.12:6823","nonce":11575}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.12:6820","nonce":11575},{"type":"v1","addr":"172.21.15.12:6821","nonce":11575}]},"public_addr":"172.21.15.12:6817/11575","cluster_addr":"172.21.15.12:6819/11575","heartbeat_back_addr":"172.21.15.12:6823/11575","heartbeat_front_addr":"172.21.15.12:6821/11575","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.334490+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.154996+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:17.391866+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:26.398 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.0 flush_pg_stats 2021-03-02T12:02:26.562 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2021-03-02T12:02:26.562 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.1 flush_pg_stats 2021-03-02T12:02:26.726 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2021-03-02T12:02:26.727 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early tell osd.2 flush_pg_stats 2021-03-02T12:02:26.893 INFO:teuthology.orchestra.run.smithi012.stdout:34359738372 2021-03-02T12:02:26.894 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-03-02T12:02:27.172 INFO:teuthology.orchestra.run.smithi012.stdout:34359738370 2021-03-02T12:02:27.191 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-03-02T12:02:28.192 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.0 2021-03-02T12:02:28.471 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2021-03-02T12:02:28.767 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-03-02T12:02:28.768 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.1 2021-03-02T12:02:29.051 INFO:teuthology.orchestra.run.smithi012.stdout:34359738371 2021-03-02T12:02:29.178 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2021-03-02T12:02:29.448 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early osd last-stat-seq osd.2 2021-03-02T12:02:29.729 INFO:teuthology.orchestra.run.smithi012.stdout:34359738372 2021-03-02T12:02:29.743 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-03-02T12:02:29.743 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-03-02T12:02:29.744 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2021-03-02T12:02:30.020 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:30.168 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2021-03-02T12:02:30.168 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-02T12:02:29.468670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.890046"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599647+0000","last_change":"2021-03-02T12:02:24.594836+0000","last_active":"2021-03-02T12:02:24.599647+0000","last_peered":"2021-03-02T12:02:24.599647+0000","last_clean":"2021-03-02T12:02:24.599647+0000","last_became_active":"2021-03-02T12:02:24.594274+0000","last_became_peered":"2021-03-02T12:02:24.594274+0000","last_unstale":"2021-03-02T12:02:24.599647+0000","last_undegraded":"2021-03-02T12:02:24.599647+0000","last_fullsized":"2021-03-02T12:02:24.599647+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599623+0000","last_change":"2021-03-02T12:02:24.594752+0000","last_active":"2021-03-02T12:02:24.599623+0000","last_peered":"2021-03-02T12:02:24.599623+0000","last_clean":"2021-03-02T12:02:24.599623+0000","last_became_active":"2021-03-02T12:02:24.594324+0000","last_became_peered":"2021-03-02T12:02:24.594324+0000","last_unstale":"2021-03-02T12:02:24.599623+0000","last_undegraded":"2021-03-02T12:02:24.599623+0000","last_fullsized":"2021-03-02T12:02:24.599623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599094+0000","last_change":"2021-03-02T12:02:24.590494+0000","last_active":"2021-03-02T12:02:24.599094+0000","last_peered":"2021-03-02T12:02:24.599094+0000","last_clean":"2021-03-02T12:02:24.599094+0000","last_became_active":"2021-03-02T12:02:24.589903+0000","last_became_peered":"2021-03-02T12:02:24.589903+0000","last_unstale":"2021-03-02T12:02:24.599094+0000","last_undegraded":"2021-03-02T12:02:24.599094+0000","last_fullsized":"2021-03-02T12:02:24.599094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.594591+0000","last_change":"2021-03-02T12:02:24.594591+0000","last_active":"2021-03-02T12:02:24.594591+0000","last_peered":"2021-03-02T12:02:24.594591+0000","last_clean":"2021-03-02T12:02:24.594591+0000","last_became_active":"2021-03-02T12:02:24.594205+0000","last_became_peered":"2021-03-02T12:02:24.594205+0000","last_unstale":"2021-03-02T12:02:24.594591+0000","last_undegraded":"2021-03-02T12:02:24.594591+0000","last_fullsized":"2021-03-02T12:02:24.594591+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.588296+0000","last_change":"2021-03-02T12:02:19.651656+0000","last_active":"2021-03-02T12:02:24.588296+0000","last_peered":"2021-03-02T12:02:24.588296+0000","last_clean":"2021-03-02T12:02:24.588296+0000","last_became_active":"2021-03-02T12:02:19.650442+0000","last_became_peered":"2021-03-02T12:02:19.650442+0000","last_unstale":"2021-03-02T12:02:24.588296+0000","last_undegraded":"2021-03-02T12:02:24.588296+0000","last_fullsized":"2021-03-02T12:02:24.588296+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:17.520643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:17.520643+0000","last_clean_scrub_stamp":"2021-03-02T12:02:17.520643+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599044+0000","last_change":"2021-03-02T12:02:24.589709+0000","last_active":"2021-03-02T12:02:24.599044+0000","last_peered":"2021-03-02T12:02:24.599044+0000","last_clean":"2021-03-02T12:02:24.599044+0000","last_became_active":"2021-03-02T12:02:24.589210+0000","last_became_peered":"2021-03-02T12:02:24.589210+0000","last_unstale":"2021-03-02T12:02:24.599044+0000","last_undegraded":"2021-03-02T12:02:24.599044+0000","last_fullsized":"2021-03-02T12:02:24.599044+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.591077+0000","last_change":"2021-03-02T12:02:24.591077+0000","last_active":"2021-03-02T12:02:24.591077+0000","last_peered":"2021-03-02T12:02:24.591077+0000","last_clean":"2021-03-02T12:02:24.591077+0000","last_became_active":"2021-03-02T12:02:24.590315+0000","last_became_peered":"2021-03-02T12:02:24.590315+0000","last_unstale":"2021-03-02T12:02:24.591077+0000","last_undegraded":"2021-03-02T12:02:24.591077+0000","last_fullsized":"2021-03-02T12:02:24.591077+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.595688+0000","last_change":"2021-03-02T12:02:24.595688+0000","last_active":"2021-03-02T12:02:24.595688+0000","last_peered":"2021-03-02T12:02:24.595688+0000","last_clean":"2021-03-02T12:02:24.595688+0000","last_became_active":"2021-03-02T12:02:24.595209+0000","last_became_peered":"2021-03-02T12:02:24.595209+0000","last_unstale":"2021-03-02T12:02:24.595688+0000","last_undegraded":"2021-03-02T12:02:24.595688+0000","last_fullsized":"2021-03-02T12:02:24.595688+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599103+0000","last_change":"2021-03-02T12:02:24.590665+0000","last_active":"2021-03-02T12:02:24.599103+0000","last_peered":"2021-03-02T12:02:24.599103+0000","last_clean":"2021-03-02T12:02:24.599103+0000","last_became_active":"2021-03-02T12:02:24.590015+0000","last_became_peered":"2021-03-02T12:02:24.590015+0000","last_unstale":"2021-03-02T12:02:24.599103+0000","last_undegraded":"2021-03-02T12:02:24.599103+0000","last_fullsized":"2021-03-02T12:02:24.599103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}]}} 2021-03-02T12:02:30.169 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early pg dump --format=json 2021-03-02T12:02:30.444 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:30.446 INFO:teuthology.orchestra.run.smithi012.stderr:dumped all 2021-03-02T12:02:30.458 INFO:teuthology.orchestra.run.smithi012.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-02T12:02:29.468670+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146072,"kb_used_data":320,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426728,"statfs":{"total":322122547200,"available":318900969472,"internally_reserved":3221225472,"allocated":327680,"data_stored":92382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.890046"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599647+0000","last_change":"2021-03-02T12:02:24.594836+0000","last_active":"2021-03-02T12:02:24.599647+0000","last_peered":"2021-03-02T12:02:24.599647+0000","last_clean":"2021-03-02T12:02:24.599647+0000","last_became_active":"2021-03-02T12:02:24.594274+0000","last_became_peered":"2021-03-02T12:02:24.594274+0000","last_unstale":"2021-03-02T12:02:24.599647+0000","last_undegraded":"2021-03-02T12:02:24.599647+0000","last_fullsized":"2021-03-02T12:02:24.599647+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599623+0000","last_change":"2021-03-02T12:02:24.594752+0000","last_active":"2021-03-02T12:02:24.599623+0000","last_peered":"2021-03-02T12:02:24.599623+0000","last_clean":"2021-03-02T12:02:24.599623+0000","last_became_active":"2021-03-02T12:02:24.594324+0000","last_became_peered":"2021-03-02T12:02:24.594324+0000","last_unstale":"2021-03-02T12:02:24.599623+0000","last_undegraded":"2021-03-02T12:02:24.599623+0000","last_fullsized":"2021-03-02T12:02:24.599623+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599094+0000","last_change":"2021-03-02T12:02:24.590494+0000","last_active":"2021-03-02T12:02:24.599094+0000","last_peered":"2021-03-02T12:02:24.599094+0000","last_clean":"2021-03-02T12:02:24.599094+0000","last_became_active":"2021-03-02T12:02:24.589903+0000","last_became_peered":"2021-03-02T12:02:24.589903+0000","last_unstale":"2021-03-02T12:02:24.599094+0000","last_undegraded":"2021-03-02T12:02:24.599094+0000","last_fullsized":"2021-03-02T12:02:24.599094+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.594591+0000","last_change":"2021-03-02T12:02:24.594591+0000","last_active":"2021-03-02T12:02:24.594591+0000","last_peered":"2021-03-02T12:02:24.594591+0000","last_clean":"2021-03-02T12:02:24.594591+0000","last_became_active":"2021-03-02T12:02:24.594205+0000","last_became_peered":"2021-03-02T12:02:24.594205+0000","last_unstale":"2021-03-02T12:02:24.594591+0000","last_undegraded":"2021-03-02T12:02:24.594591+0000","last_fullsized":"2021-03-02T12:02:24.594591+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.588296+0000","last_change":"2021-03-02T12:02:19.651656+0000","last_active":"2021-03-02T12:02:24.588296+0000","last_peered":"2021-03-02T12:02:24.588296+0000","last_clean":"2021-03-02T12:02:24.588296+0000","last_became_active":"2021-03-02T12:02:19.650442+0000","last_became_peered":"2021-03-02T12:02:19.650442+0000","last_unstale":"2021-03-02T12:02:24.588296+0000","last_undegraded":"2021-03-02T12:02:24.588296+0000","last_fullsized":"2021-03-02T12:02:24.588296+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:17.520643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:17.520643+0000","last_clean_scrub_stamp":"2021-03-02T12:02:17.520643+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599044+0000","last_change":"2021-03-02T12:02:24.589709+0000","last_active":"2021-03-02T12:02:24.599044+0000","last_peered":"2021-03-02T12:02:24.599044+0000","last_clean":"2021-03-02T12:02:24.599044+0000","last_became_active":"2021-03-02T12:02:24.589210+0000","last_became_peered":"2021-03-02T12:02:24.589210+0000","last_unstale":"2021-03-02T12:02:24.599044+0000","last_undegraded":"2021-03-02T12:02:24.599044+0000","last_fullsized":"2021-03-02T12:02:24.599044+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.591077+0000","last_change":"2021-03-02T12:02:24.591077+0000","last_active":"2021-03-02T12:02:24.591077+0000","last_peered":"2021-03-02T12:02:24.591077+0000","last_clean":"2021-03-02T12:02:24.591077+0000","last_became_active":"2021-03-02T12:02:24.590315+0000","last_became_peered":"2021-03-02T12:02:24.590315+0000","last_unstale":"2021-03-02T12:02:24.591077+0000","last_undegraded":"2021-03-02T12:02:24.591077+0000","last_fullsized":"2021-03-02T12:02:24.591077+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.595688+0000","last_change":"2021-03-02T12:02:24.595688+0000","last_active":"2021-03-02T12:02:24.595688+0000","last_peered":"2021-03-02T12:02:24.595688+0000","last_clean":"2021-03-02T12:02:24.595688+0000","last_became_active":"2021-03-02T12:02:24.595209+0000","last_became_peered":"2021-03-02T12:02:24.595209+0000","last_unstale":"2021-03-02T12:02:24.595688+0000","last_undegraded":"2021-03-02T12:02:24.595688+0000","last_fullsized":"2021-03-02T12:02:24.595688+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:24.599103+0000","last_change":"2021-03-02T12:02:24.590665+0000","last_active":"2021-03-02T12:02:24.599103+0000","last_peered":"2021-03-02T12:02:24.599103+0000","last_clean":"2021-03-02T12:02:24.599103+0000","last_became_active":"2021-03-02T12:02:24.590015+0000","last_became_peered":"2021-03-02T12:02:24.590015+0000","last_unstale":"2021-03-02T12:02:24.599103+0000","last_undegraded":"2021-03-02T12:02:24.599103+0000","last_fullsized":"2021-03-02T12:02:24.599103+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:23.565879+0000","last_clean_scrub_stamp":"2021-03-02T12:02:23.565879+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54600000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67600000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]}]}]}} 2021-03-02T12:02:30.459 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-03-02T12:02:30.459 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-03-02T12:02:30.460 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-03-02T12:02:30.460 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 --log-early health --format=json 2021-03-02T12:02:30.782 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:31.021 INFO:teuthology.orchestra.run.smithi012.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-02T12:02:31.022 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-03-02T12:02:31.022 INFO:teuthology.run_tasks:Running task ceph... 2021-03-02T12:02:31.039 INFO:tasks.ceph:Creating ceph cluster c2... 2021-03-02T12:02:31.040 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-02T12:02:31.041 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi/5925536', 'branch': 'wip-yuri-testing-2021-03-01-1125-octopus', 'description': 'rgw/multisite/{clusters frontend/beast omap_limits overrides realms/three-zone-plus-pubsub tasks/test_multi valgrind}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '5925536', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2021-03-01_21:06:29-rgw-wip-yuri-testing-2021-03-01-1125-octopus-distro-basic-smithi', 'nuke-on-error': True, 'os_type': 'ubuntu', 'overrides': {'admin_socket': {'branch': 'wip-yuri-testing-2021-03-01-1125-octopus'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '3664024eebddedeee285ff2f143b16556af4e85d'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c1.client.1'], 'name': 'test-zone3'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone4'}]}]}, 'workunit': {'branch': 'wip-yuri-testing-2021-03-01-1125-octopus', 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 100, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0', 'c1.client.1'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'sleep_before_teardown': 0, 'suite': 'rgw', 'suite_branch': 'wip-yuri-testing-2021-03-01-1125-octopus', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_3664024eebddedeee285ff2f143b16556af4e85d/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'targets': {'smithi012.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8QYmvqWVKZIAMULP+iGClqAj+x24U+MP7H3fZe4odr07HfPHg/CnVtjfdT2QyGoU5VWBGYl+SEuXm0F9MLifQPqrx89kTr4LayNySLJQrh+hjJta+XAB+16HRRmbzsRs54FclMEAO+Ogu18SUyjovXIFKv3EztPP8YYLZZ/0jwJPQUDmu1dL8y9dVg7DeVO3oVI2TxHwNoqeYdEedr35bO2P5fJZ2u11BU5j2X0gMI4J6IhdUplsGqUf2MSnMtXrod9tIteE5Qh9Kq8uBdbatCDaRYEULY9sk8PahmGsDPxGeg0gU+TKDKvpUXx2vgVkByYO4eOW0k4zylvLEhi9h', 'smithi036.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIsrda0walO1WCtov3w5mYxsSoKKIQyKQdRRtrtL3bxQ0LNiCw/gthawcXiWIIG8Nh27MjSrXMSVKshdoXMQdf74860hF1Wrpc27wEmlEZgdly3AgAnyxTqvHcORqQLj4p7XZbiwk6uKsbXjWaEdrYGYAPt9/D6CIazdKZKRbAs/04UE4xeS/66T6cnuYPGV96mPMpyCN4FdBTvg7nOARpckzH2UkDMtOjFiT6ebPowrJeTt4qawjPfEcsG4ZeWqQPsI/xqcF3EXWoYd1Rmrobof2UL1bZ7KLD7Fl6tA6oyl1330KFYcKF9wmhCFZnqSW/BYmjtuVkDTWDwhafLP3J'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'global': {'osd heartbeat grace': 40}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon osd crush smoke test': False}, 'osd': {'debug filestore': 20, 'debug journal': 20, 'debug ms': 1, 'debug osd': 25, 'osd fast shutdown': False, 'rocksdb delete range threshold': 0}, 'osd.0': {'osd_max_omap_entries_per_request': 10}, 'osd.1': {'osd_max_omap_entries_per_request': 1000}, 'osd.2': {'osd_max_omap_entries_per_request': 10000}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '3664024eebddedeee285ff2f143b16556af4e85d', 'wait-for-scrub': False}}, {'rgw': {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '3082387bba74fcd24c9700593d10418152d53c97', 'timestamp': '2021-03-01_21:06:29', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.25973'} 2021-03-02T12:02:31.042 DEBUG:teuthology.orchestra.run.smithi012:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-03-02T12:02:31.046 DEBUG:teuthology.orchestra.run.smithi036:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-03-02T12:02:31.055 DEBUG:teuthology.orchestra.run.smithi012:> sudo install -d -m0777 -- /var/run/ceph 2021-03-02T12:02:31.095 DEBUG:teuthology.orchestra.run.smithi036:> sudo install -d -m0777 -- /var/run/ceph 2021-03-02T12:02:31.114 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:31.115 DEBUG:teuthology.orchestra.run.smithi036:> dd if=/scratch_devs of=/dev/stdout 2021-03-02T12:02:31.162 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-02T12:02:31.162 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-02T12:02:31.210 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-02T12:02:31.210 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:31.211 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-03-02T12:02:31.211 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:31.211 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2021-03-02 11:59:21.530490141 +0000 2021-03-02T12:02:31.212 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2021-03-02 11:59:21.414490118 +0000 2021-03-02T12:02:31.212 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2021-03-02 11:59:21.414490118 +0000 2021-03-02T12:02:31.213 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2021-03-02T12:02:31.220 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2021-03-02T12:02:31.220 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2021-03-02T12:02:31.221 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000401451 s, 1.3 MB/s 2021-03-02T12:02:31.225 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-02T12:02:31.274 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-02T12:02:31.274 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:31.274 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-03-02T12:02:31.275 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:31.275 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2021-03-02 11:59:21.806490195 +0000 2021-03-02T12:02:31.276 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2021-03-02 11:59:21.678490170 +0000 2021-03-02T12:02:31.276 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2021-03-02 11:59:21.678490170 +0000 2021-03-02T12:02:31.276 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2021-03-02T12:02:31.283 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2021-03-02T12:02:31.283 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2021-03-02T12:02:31.284 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000302389 s, 1.7 MB/s 2021-03-02T12:02:31.287 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-02T12:02:31.338 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-02T12:02:31.495 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:31.496 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 356 Links: 1 2021-03-02T12:02:31.496 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:31.497 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2021-03-02 11:59:22.114490256 +0000 2021-03-02T12:02:31.497 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2021-03-02 11:59:21.934490221 +0000 2021-03-02T12:02:31.498 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2021-03-02 11:59:21.934490221 +0000 2021-03-02T12:02:31.498 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2021-03-02T12:02:31.500 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2021-03-02T12:02:31.501 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2021-03-02T12:02:31.501 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000337405 s, 1.5 MB/s 2021-03-02T12:02:31.502 DEBUG:teuthology.orchestra.run.smithi036:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-02T12:02:31.549 INFO:teuthology.orchestra.run.smithi036.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-02T12:02:31.550 INFO:teuthology.orchestra.run.smithi036.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-02T12:02:31.550 INFO:teuthology.orchestra.run.smithi036.stdout:Device: 6h/6d Inode: 363 Links: 1 2021-03-02T12:02:31.550 INFO:teuthology.orchestra.run.smithi036.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-02T12:02:31.550 INFO:teuthology.orchestra.run.smithi036.stdout:Access: 2021-03-02 11:59:22.254490284 +0000 2021-03-02T12:02:31.551 INFO:teuthology.orchestra.run.smithi036.stdout:Modify: 2021-03-02 11:59:22.250490283 +0000 2021-03-02T12:02:31.551 INFO:teuthology.orchestra.run.smithi036.stdout:Change: 2021-03-02 11:59:22.250490283 +0000 2021-03-02T12:02:31.551 INFO:teuthology.orchestra.run.smithi036.stdout: Birth: - 2021-03-02T12:02:31.560 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records in 2021-03-02T12:02:31.561 INFO:teuthology.orchestra.run.smithi036.stderr:1+0 records out 2021-03-02T12:02:31.561 INFO:teuthology.orchestra.run.smithi036.stderr:512 bytes copied, 0.000420744 s, 1.2 MB/s 2021-03-02T12:02:31.564 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:31.565 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi036.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-02T12:02:31.565 INFO:tasks.ceph:Generating config... 2021-03-02T12:02:31.571 INFO:tasks.ceph:[client] debug rgw = 20 2021-03-02T12:02:31.571 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-03-02T12:02:31.572 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-03-02T12:02:31.572 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-02T12:02:31.573 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-03-02T12:02:31.574 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-03-02T12:02:31.574 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-03-02T12:02:31.575 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-03-02T12:02:31.576 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-03-02T12:02:31.576 INFO:tasks.ceph:[client] setgroup = ceph 2021-03-02T12:02:31.577 INFO:tasks.ceph:[client] setuser = ceph 2021-03-02T12:02:31.577 INFO:tasks.ceph:[global] osd heartbeat grace = 40 2021-03-02T12:02:31.578 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-02T12:02:31.578 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-02T12:02:31.579 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-02T12:02:31.579 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-02T12:02:31.580 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-02T12:02:31.581 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2021-03-02T12:02:31.581 INFO:tasks.ceph:[osd] debug filestore = 20 2021-03-02T12:02:31.582 INFO:tasks.ceph:[osd] debug journal = 20 2021-03-02T12:02:31.582 INFO:tasks.ceph:[osd] debug ms = 1 2021-03-02T12:02:31.583 INFO:tasks.ceph:[osd] debug osd = 25 2021-03-02T12:02:31.583 INFO:tasks.ceph:[osd] osd fast shutdown = False 2021-03-02T12:02:31.584 INFO:tasks.ceph:[osd] rocksdb delete range threshold = 0 2021-03-02T12:02:31.584 INFO:tasks.ceph:[osd.0] osd_max_omap_entries_per_request = 10 2021-03-02T12:02:31.585 INFO:tasks.ceph:[osd.1] osd_max_omap_entries_per_request = 1000 2021-03-02T12:02:31.586 INFO:tasks.ceph:[osd.2] osd_max_omap_entries_per_request = 10000 2021-03-02T12:02:31.586 INFO:tasks.ceph:Setting up c2.mon.a... 2021-03-02T12:02:31.587 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-03-02T12:02:31.639 INFO:teuthology.orchestra.run.smithi036.stdout:creating /etc/ceph/c2.keyring 2021-03-02T12:02:31.640 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-03-02T12:02:31.724 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-02T12:02:32.080 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.36')] 2021-03-02T12:02:32.081 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.36 --print /home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:32.150 INFO:teuthology.orchestra.run.smithi036.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:32.151 INFO:teuthology.orchestra.run.smithi036.stdout:monmaptool: generated fsid c58b7310-abb3-4c11-b73a-9b0fb0709dc8 2021-03-02T12:02:32.151 INFO:teuthology.orchestra.run.smithi036.stdout:epoch 0 2021-03-02T12:02:32.152 INFO:teuthology.orchestra.run.smithi036.stdout:fsid c58b7310-abb3-4c11-b73a-9b0fb0709dc8 2021-03-02T12:02:32.152 INFO:teuthology.orchestra.run.smithi036.stdout:last_changed 2021-03-02T12:02:32.151826+0000 2021-03-02T12:02:32.153 INFO:teuthology.orchestra.run.smithi036.stdout:created 2021-03-02T12:02:32.151826+0000 2021-03-02T12:02:32.153 INFO:teuthology.orchestra.run.smithi036.stdout:min_mon_release 0 (unknown) 2021-03-02T12:02:32.153 INFO:teuthology.orchestra.run.smithi036.stdout:0: [v2:172.21.15.36:3300/0,v1:172.21.15.36:6789/0] mon.a 2021-03-02T12:02:32.153 INFO:teuthology.orchestra.run.smithi036.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-03-02T12:02:32.155 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID c58b7310-abb3-4c11-b73a-9b0fb0709dc8... 2021-03-02T12:02:32.156 DEBUG:teuthology.orchestra.run.smithi012:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-03-02T12:02:32.161 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-03-02T12:02:32.202 INFO:teuthology.orchestra.run.smithi012.stdout:[global] 2021-03-02T12:02:32.202 INFO:teuthology.orchestra.run.smithi012.stdout: chdir = "" 2021-03-02T12:02:32.203 INFO:teuthology.orchestra.run.smithi012.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-02T12:02:32.203 INFO:teuthology.orchestra.run.smithi012.stdout: auth supported = cephx 2021-03-02T12:02:32.203 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.204 INFO:teuthology.orchestra.run.smithi012.stdout: filestore xattr use omap = true 2021-03-02T12:02:32.204 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.204 INFO:teuthology.orchestra.run.smithi012.stdout: mon clock drift allowed = 1.000 2021-03-02T12:02:32.204 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.205 INFO:teuthology.orchestra.run.smithi012.stdout: osd crush chooseleaf type = 0 2021-03-02T12:02:32.205 INFO:teuthology.orchestra.run.smithi012.stdout: auth debug = true 2021-03-02T12:02:32.205 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.206 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on old message = true 2021-03-02T12:02:32.206 INFO:teuthology.orchestra.run.smithi012.stdout: ms die on bug = true 2021-03-02T12:02:32.206 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.206 INFO:teuthology.orchestra.run.smithi012.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-02T12:02:32.207 INFO:teuthology.orchestra.run.smithi012.stdout: mon pg warn max object skew = 0 2021-03-02T12:02:32.207 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.207 INFO:teuthology.orchestra.run.smithi012.stdout: # disable pg_autoscaler by default for new pools 2021-03-02T12:02:32.208 INFO:teuthology.orchestra.run.smithi012.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-02T12:02:32.208 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.208 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default size = 2 2021-03-02T12:02:32.209 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.209 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow primary affinity = true 2021-03-02T12:02:32.210 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd allow pg remap = true 2021-03-02T12:02:32.210 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on legacy crush tunables = false 2021-03-02T12:02:32.210 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on crush straw calc version zero = false 2021-03-02T12:02:32.211 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on no sortbitwise = false 2021-03-02T12:02:32.211 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on osd down out interval zero = false 2021-03-02T12:02:32.211 INFO:teuthology.orchestra.run.smithi012.stdout: mon warn on too few osds = false 2021-03-02T12:02:32.211 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-02T12:02:32.211 INFO:teuthology.orchestra.run.smithi012.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-02T12:02:32.212 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.212 INFO:teuthology.orchestra.run.smithi012.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-02T12:02:32.212 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.212 INFO:teuthology.orchestra.run.smithi012.stdout: osd default data pool replay window = 5 2021-03-02T12:02:32.212 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.213 INFO:teuthology.orchestra.run.smithi012.stdout: mon allow pool delete = true 2021-03-02T12:02:32.213 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.213 INFO:teuthology.orchestra.run.smithi012.stdout: mon cluster log file level = debug 2021-03-02T12:02:32.213 INFO:teuthology.orchestra.run.smithi012.stdout: debug asserts on shutdown = true 2021-03-02T12:02:32.214 INFO:teuthology.orchestra.run.smithi012.stdout: mon health detail to clog = false 2021-03-02T12:02:32.214 INFO:teuthology.orchestra.run.smithi012.stdout: mon host = 172.21.15.36 2021-03-02T12:02:32.214 INFO:teuthology.orchestra.run.smithi012.stdout: osd heartbeat grace = 40 2021-03-02T12:02:32.214 INFO:teuthology.orchestra.run.smithi012.stdout: fsid = c58b7310-abb3-4c11-b73a-9b0fb0709dc8 2021-03-02T12:02:32.214 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.215 INFO:teuthology.orchestra.run.smithi012.stdout:[osd] 2021-03-02T12:02:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: osd journal size = 100 2021-03-02T12:02:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub load threshold = 5.0 2021-03-02T12:02:32.215 INFO:teuthology.orchestra.run.smithi012.stdout: osd scrub max interval = 600 2021-03-02T12:02:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: osd recover clone overlap = true 2021-03-02T12:02:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: osd recovery max chunk = 1048576 2021-03-02T12:02:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.216 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug shutdown = true 2021-03-02T12:02:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug op order = true 2021-03-02T12:02:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug verify stray on activate = true 2021-03-02T12:02:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: osd open classes on start = true 2021-03-02T12:02:32.217 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug pg log writeout = true 2021-03-02T12:02:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: osd deep scrub update digest min age = 30 2021-03-02T12:02:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.218 INFO:teuthology.orchestra.run.smithi012.stdout: osd map max advance = 10 2021-03-02T12:02:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: journal zero on create = true 2021-03-02T12:02:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: filestore ondisk finisher threads = 3 2021-03-02T12:02:32.219 INFO:teuthology.orchestra.run.smithi012.stdout: filestore apply finisher threads = 3 2021-03-02T12:02:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: bdev debug aio = true 2021-03-02T12:02:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: osd debug misdirected ops = true 2021-03-02T12:02:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: debug filestore = 20 2021-03-02T12:02:32.220 INFO:teuthology.orchestra.run.smithi012.stdout: debug journal = 20 2021-03-02T12:02:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2021-03-02T12:02:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: debug osd = 25 2021-03-02T12:02:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: osd fast shutdown = False 2021-03-02T12:02:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: rocksdb delete range threshold = 0 2021-03-02T12:02:32.221 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.222 INFO:teuthology.orchestra.run.smithi012.stdout:[mgr] 2021-03-02T12:02:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2021-03-02T12:02:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: debug mgr = 20 2021-03-02T12:02:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2021-03-02T12:02:32.222 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2021-03-02T12:02:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: mgr/telemetry/nag = false 2021-03-02T12:02:32.223 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.223 INFO:teuthology.orchestra.run.smithi012.stdout:[mon] 2021-03-02T12:02:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: debug ms = 1 2021-03-02T12:02:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: debug mon = 20 2021-03-02T12:02:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: debug paxos = 20 2021-03-02T12:02:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: debug auth = 20 2021-03-02T12:02:32.224 INFO:teuthology.orchestra.run.smithi012.stdout: mon data avail warn = 5 2021-03-02T12:02:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: mon mgr mkfs grace = 240 2021-03-02T12:02:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd reporter subtree level = osd 2021-03-02T12:02:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd prime pg temp = true 2021-03-02T12:02:32.225 INFO:teuthology.orchestra.run.smithi012.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: mon osd crush smoke test = False 2021-03-02T12:02:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: 2021-03-02T12:02:32.226 INFO:teuthology.orchestra.run.smithi012.stdout:[client] 2021-03-02T12:02:32.226 INFO:teuthology.orchestra.run.smithi012.stdout: rgw cache enabled = true 2021-03-02T12:02:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable ops log = true 2021-03-02T12:02:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: rgw enable usage log = true 2021-03-02T12:02:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-02T12:02:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-02T12:02:32.227 INFO:teuthology.orchestra.run.smithi012.stdout: debug rgw = 20 2021-03-02T12:02:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt require ssl = False 2021-03-02T12:02:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms backend = testing 2021-03-02T12:02:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-02T12:02:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: rgw curl low speed time = 300 2021-03-02T12:02:32.228 INFO:teuthology.orchestra.run.smithi012.stdout: rgw data log num shards = 4 2021-03-02T12:02:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: rgw md log max shards = 4 2021-03-02T12:02:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync log trim interval = 0 2021-03-02T12:02:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: rgw sync obj etag verify = True 2021-03-02T12:02:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: setgroup = ceph 2021-03-02T12:02:32.229 INFO:teuthology.orchestra.run.smithi012.stdout: setuser = ceph 2021-03-02T12:02:32.230 INFO:teuthology.orchestra.run.smithi012.stdout:[mon.a] 2021-03-02T12:02:32.230 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.0] 2021-03-02T12:02:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10 2021-03-02T12:02:32.230 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.1] 2021-03-02T12:02:32.230 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 1000 2021-03-02T12:02:32.231 INFO:teuthology.orchestra.run.smithi012.stdout:[osd.2] 2021-03-02T12:02:32.231 INFO:teuthology.orchestra.run.smithi012.stdout: osd_max_omap_entries_per_request = 10000 2021-03-02T12:02:32.232 INFO:teuthology.orchestra.run.smithi036.stdout:[global] 2021-03-02T12:02:32.232 INFO:teuthology.orchestra.run.smithi036.stdout: chdir = "" 2021-03-02T12:02:32.232 INFO:teuthology.orchestra.run.smithi036.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-02T12:02:32.233 INFO:teuthology.orchestra.run.smithi036.stdout: auth supported = cephx 2021-03-02T12:02:32.233 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.233 INFO:teuthology.orchestra.run.smithi036.stdout: filestore xattr use omap = true 2021-03-02T12:02:32.233 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.233 INFO:teuthology.orchestra.run.smithi036.stdout: mon clock drift allowed = 1.000 2021-03-02T12:02:32.234 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.234 INFO:teuthology.orchestra.run.smithi036.stdout: osd crush chooseleaf type = 0 2021-03-02T12:02:32.234 INFO:teuthology.orchestra.run.smithi036.stdout: auth debug = true 2021-03-02T12:02:32.234 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.234 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on old message = true 2021-03-02T12:02:32.235 INFO:teuthology.orchestra.run.smithi036.stdout: ms die on bug = true 2021-03-02T12:02:32.235 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.235 INFO:teuthology.orchestra.run.smithi036.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-02T12:02:32.235 INFO:teuthology.orchestra.run.smithi036.stdout: mon pg warn max object skew = 0 2021-03-02T12:02:32.236 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.236 INFO:teuthology.orchestra.run.smithi036.stdout: # disable pg_autoscaler by default for new pools 2021-03-02T12:02:32.236 INFO:teuthology.orchestra.run.smithi036.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-02T12:02:32.236 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.236 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default size = 2 2021-03-02T12:02:32.237 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow primary affinity = true 2021-03-02T12:02:32.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd allow pg remap = true 2021-03-02T12:02:32.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on legacy crush tunables = false 2021-03-02T12:02:32.237 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on crush straw calc version zero = false 2021-03-02T12:02:32.238 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on no sortbitwise = false 2021-03-02T12:02:32.238 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on osd down out interval zero = false 2021-03-02T12:02:32.238 INFO:teuthology.orchestra.run.smithi036.stdout: mon warn on too few osds = false 2021-03-02T12:02:32.238 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-02T12:02:32.238 INFO:teuthology.orchestra.run.smithi036.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-02T12:02:32.239 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.239 INFO:teuthology.orchestra.run.smithi036.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-02T12:02:32.239 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.239 INFO:teuthology.orchestra.run.smithi036.stdout: osd default data pool replay window = 5 2021-03-02T12:02:32.239 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.240 INFO:teuthology.orchestra.run.smithi036.stdout: mon allow pool delete = true 2021-03-02T12:02:32.240 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.240 INFO:teuthology.orchestra.run.smithi036.stdout: mon cluster log file level = debug 2021-03-02T12:02:32.240 INFO:teuthology.orchestra.run.smithi036.stdout: debug asserts on shutdown = true 2021-03-02T12:02:32.241 INFO:teuthology.orchestra.run.smithi036.stdout: mon health detail to clog = false 2021-03-02T12:02:32.241 INFO:teuthology.orchestra.run.smithi036.stdout: mon host = 172.21.15.36 2021-03-02T12:02:32.241 INFO:teuthology.orchestra.run.smithi036.stdout: osd heartbeat grace = 40 2021-03-02T12:02:32.241 INFO:teuthology.orchestra.run.smithi036.stdout: fsid = c58b7310-abb3-4c11-b73a-9b0fb0709dc8 2021-03-02T12:02:32.241 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.242 INFO:teuthology.orchestra.run.smithi036.stdout:[osd] 2021-03-02T12:02:32.242 INFO:teuthology.orchestra.run.smithi036.stdout: osd journal size = 100 2021-03-02T12:02:32.242 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.242 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub load threshold = 5.0 2021-03-02T12:02:32.242 INFO:teuthology.orchestra.run.smithi036.stdout: osd scrub max interval = 600 2021-03-02T12:02:32.243 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.243 INFO:teuthology.orchestra.run.smithi036.stdout: osd recover clone overlap = true 2021-03-02T12:02:32.243 INFO:teuthology.orchestra.run.smithi036.stdout: osd recovery max chunk = 1048576 2021-03-02T12:02:32.243 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.243 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug shutdown = true 2021-03-02T12:02:32.244 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug op order = true 2021-03-02T12:02:32.244 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug verify stray on activate = true 2021-03-02T12:02:32.244 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.244 INFO:teuthology.orchestra.run.smithi036.stdout: osd open classes on start = true 2021-03-02T12:02:32.244 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug pg log writeout = true 2021-03-02T12:02:32.245 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.245 INFO:teuthology.orchestra.run.smithi036.stdout: osd deep scrub update digest min age = 30 2021-03-02T12:02:32.245 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.245 INFO:teuthology.orchestra.run.smithi036.stdout: osd map max advance = 10 2021-03-02T12:02:32.245 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.246 INFO:teuthology.orchestra.run.smithi036.stdout: journal zero on create = true 2021-03-02T12:02:32.246 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.246 INFO:teuthology.orchestra.run.smithi036.stdout: filestore ondisk finisher threads = 3 2021-03-02T12:02:32.246 INFO:teuthology.orchestra.run.smithi036.stdout: filestore apply finisher threads = 3 2021-03-02T12:02:32.246 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.247 INFO:teuthology.orchestra.run.smithi036.stdout: bdev debug aio = true 2021-03-02T12:02:32.247 INFO:teuthology.orchestra.run.smithi036.stdout: osd debug misdirected ops = true 2021-03-02T12:02:32.247 INFO:teuthology.orchestra.run.smithi036.stdout: debug filestore = 20 2021-03-02T12:02:32.247 INFO:teuthology.orchestra.run.smithi036.stdout: debug journal = 20 2021-03-02T12:02:32.247 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2021-03-02T12:02:32.248 INFO:teuthology.orchestra.run.smithi036.stdout: debug osd = 25 2021-03-02T12:02:32.248 INFO:teuthology.orchestra.run.smithi036.stdout: osd fast shutdown = False 2021-03-02T12:02:32.248 INFO:teuthology.orchestra.run.smithi036.stdout: rocksdb delete range threshold = 0 2021-03-02T12:02:32.248 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.248 INFO:teuthology.orchestra.run.smithi036.stdout:[mgr] 2021-03-02T12:02:32.249 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2021-03-02T12:02:32.249 INFO:teuthology.orchestra.run.smithi036.stdout: debug mgr = 20 2021-03-02T12:02:32.249 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2021-03-02T12:02:32.249 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2021-03-02T12:02:32.250 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:32.250 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:32.250 INFO:teuthology.orchestra.run.smithi036.stdout: mgr/telemetry/nag = false 2021-03-02T12:02:32.250 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.250 INFO:teuthology.orchestra.run.smithi036.stdout:[mon] 2021-03-02T12:02:32.251 INFO:teuthology.orchestra.run.smithi036.stdout: debug ms = 1 2021-03-02T12:02:32.251 INFO:teuthology.orchestra.run.smithi036.stdout: debug mon = 20 2021-03-02T12:02:32.251 INFO:teuthology.orchestra.run.smithi036.stdout: debug paxos = 20 2021-03-02T12:02:32.251 INFO:teuthology.orchestra.run.smithi036.stdout: debug auth = 20 2021-03-02T12:02:32.251 INFO:teuthology.orchestra.run.smithi036.stdout: mon data avail warn = 5 2021-03-02T12:02:32.252 INFO:teuthology.orchestra.run.smithi036.stdout: mon mgr mkfs grace = 240 2021-03-02T12:02:32.252 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min pgs per osd = 4 2021-03-02T12:02:32.252 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd reporter subtree level = osd 2021-03-02T12:02:32.252 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd prime pg temp = true 2021-03-02T12:02:32.252 INFO:teuthology.orchestra.run.smithi036.stdout: mon reweight min bytes per osd = 10 2021-03-02T12:02:32.253 INFO:teuthology.orchestra.run.smithi036.stdout: mon osd crush smoke test = False 2021-03-02T12:02:32.253 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:32.253 INFO:teuthology.orchestra.run.smithi036.stdout:[client] 2021-03-02T12:02:32.253 INFO:teuthology.orchestra.run.smithi036.stdout: rgw cache enabled = true 2021-03-02T12:02:32.253 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable ops log = true 2021-03-02T12:02:32.254 INFO:teuthology.orchestra.run.smithi036.stdout: rgw enable usage log = true 2021-03-02T12:02:32.254 INFO:teuthology.orchestra.run.smithi036.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-02T12:02:32.254 INFO:teuthology.orchestra.run.smithi036.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-02T12:02:32.254 INFO:teuthology.orchestra.run.smithi036.stdout: debug rgw = 20 2021-03-02T12:02:32.254 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt require ssl = False 2021-03-02T12:02:32.255 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms backend = testing 2021-03-02T12:02:32.255 INFO:teuthology.orchestra.run.smithi036.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-02T12:02:32.255 INFO:teuthology.orchestra.run.smithi036.stdout: rgw curl low speed time = 300 2021-03-02T12:02:32.255 INFO:teuthology.orchestra.run.smithi036.stdout: rgw data log num shards = 4 2021-03-02T12:02:32.256 INFO:teuthology.orchestra.run.smithi036.stdout: rgw md log max shards = 4 2021-03-02T12:02:32.256 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync log trim interval = 0 2021-03-02T12:02:32.256 INFO:teuthology.orchestra.run.smithi036.stdout: rgw sync obj etag verify = True 2021-03-02T12:02:32.256 INFO:teuthology.orchestra.run.smithi036.stdout: setgroup = ceph 2021-03-02T12:02:32.256 INFO:teuthology.orchestra.run.smithi036.stdout: setuser = ceph 2021-03-02T12:02:32.257 INFO:teuthology.orchestra.run.smithi036.stdout:[mon.a] 2021-03-02T12:02:32.257 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.0] 2021-03-02T12:02:32.257 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10 2021-03-02T12:02:32.257 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.1] 2021-03-02T12:02:32.257 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 1000 2021-03-02T12:02:32.258 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.2] 2021-03-02T12:02:32.258 INFO:teuthology.orchestra.run.smithi036.stdout: osd_max_omap_entries_per_request = 10000 2021-03-02T12:02:32.264 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-03-02T12:02:32.265 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-03-02T12:02:32.353 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-02T12:02:32.426 DEBUG:teuthology.orchestra.remote:smithi036:/etc/ceph/c2.keyring is 206B 2021-03-02T12:02:32.495 DEBUG:teuthology.orchestra.remote:smithi036:/home/ubuntu/cephtest/c2.monmap is 179B 2021-03-02T12:02:32.508 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi012.front.sepia.ceph.com 2021-03-02T12:02:32.508 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:32.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo dd of=/etc/ceph/c2.keyring 2021-03-02T12:02:32.509 DEBUG:teuthology.orchestra.run.smithi012:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-02T12:02:32.531 DEBUG:teuthology.orchestra.run.smithi012:> set -ex 2021-03-02T12:02:32.531 DEBUG:teuthology.orchestra.run.smithi012:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:32.580 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi036.front.sepia.ceph.com 2021-03-02T12:02:32.580 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:32.581 DEBUG:teuthology.orchestra.run.smithi036:> sudo dd of=/etc/ceph/c2.keyring 2021-03-02T12:02:32.581 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-02T12:02:32.607 DEBUG:teuthology.orchestra.run.smithi036:> set -ex 2021-03-02T12:02:32.607 DEBUG:teuthology.orchestra.run.smithi036:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:32.657 INFO:tasks.ceph:Setting up mon nodes... 2021-03-02T12:02:32.658 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-02T12:02:32.658 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-03-02T12:02:32.743 INFO:teuthology.orchestra.run.smithi036.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-03-02T12:02:33.077 INFO:tasks.ceph:Setting up mds nodes... 2021-03-02T12:02:33.078 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-02T12:02:33.079 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-03-02T12:02:33.162 INFO:teuthology.orchestra.run.smithi036.stdout:creating /etc/ceph/c2.client.0.keyring 2021-03-02T12:02:33.803 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-03-02T12:02:33.884 INFO:teuthology.orchestra.run.smithi036.stdout:creating /etc/ceph/c2.client.1.keyring 2021-03-02T12:02:33.904 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-02T12:02:33.905 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi012.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi036.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-02T12:02:33.911 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-03-02T12:02:33.966 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:33.967 INFO:tasks.ceph:role: c2.osd.0 2021-03-02T12:02:33.967 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi036.front.sepia.ceph.com 2021-03-02T12:02:33.968 DEBUG:teuthology.orchestra.run.smithi036:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-02T12:02:34.479 INFO:teuthology.orchestra.run.smithi036.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-02T12:02:34.479 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-02T12:02:34.480 INFO:teuthology.orchestra.run.smithi036.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-02T12:02:34.480 INFO:teuthology.orchestra.run.smithi036.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-02T12:02:34.481 INFO:teuthology.orchestra.run.smithi036.stdout: = sunit=0 swidth=0 blks 2021-03-02T12:02:34.481 INFO:teuthology.orchestra.run.smithi036.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-02T12:02:34.481 INFO:teuthology.orchestra.run.smithi036.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-02T12:02:34.482 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-02T12:02:34.482 INFO:teuthology.orchestra.run.smithi036.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-02T12:02:34.484 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi036.front.sepia.ceph.com -o noatime 2021-03-02T12:02:34.485 DEBUG:teuthology.orchestra.run.smithi036:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-03-02T12:02:34.568 DEBUG:teuthology.orchestra.run.smithi036:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-03-02T12:02:34.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-02T12:02:34.583 INFO:teuthology.orchestra.run.smithi036.stderr:sudo: /sbin/restorecon: command not found 2021-03-02T12:02:34.583 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-03-02T12:02:34.646 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:34.646 INFO:tasks.ceph:role: c2.osd.1 2021-03-02T12:02:34.647 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi036.front.sepia.ceph.com 2021-03-02T12:02:34.647 DEBUG:teuthology.orchestra.run.smithi036:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-02T12:02:35.161 INFO:teuthology.orchestra.run.smithi036.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-02T12:02:35.162 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-02T12:02:35.162 INFO:teuthology.orchestra.run.smithi036.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-02T12:02:35.163 INFO:teuthology.orchestra.run.smithi036.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-02T12:02:35.163 INFO:teuthology.orchestra.run.smithi036.stdout: = sunit=0 swidth=0 blks 2021-03-02T12:02:35.164 INFO:teuthology.orchestra.run.smithi036.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-02T12:02:35.164 INFO:teuthology.orchestra.run.smithi036.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-02T12:02:35.164 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-02T12:02:35.165 INFO:teuthology.orchestra.run.smithi036.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-02T12:02:35.167 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi036.front.sepia.ceph.com -o noatime 2021-03-02T12:02:35.168 DEBUG:teuthology.orchestra.run.smithi036:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-03-02T12:02:35.227 DEBUG:teuthology.orchestra.run.smithi036:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-03-02T12:02:35.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-02T12:02:35.248 INFO:teuthology.orchestra.run.smithi036.stderr:sudo: /sbin/restorecon: command not found 2021-03-02T12:02:35.249 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-03-02T12:02:35.315 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-02T12:02:35.316 INFO:tasks.ceph:role: c2.osd.2 2021-03-02T12:02:35.316 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi036.front.sepia.ceph.com 2021-03-02T12:02:35.317 DEBUG:teuthology.orchestra.run.smithi036:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-02T12:02:35.823 INFO:teuthology.orchestra.run.smithi036.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-02T12:02:35.823 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-02T12:02:35.824 INFO:teuthology.orchestra.run.smithi036.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-02T12:02:35.824 INFO:teuthology.orchestra.run.smithi036.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-02T12:02:35.824 INFO:teuthology.orchestra.run.smithi036.stdout: = sunit=0 swidth=0 blks 2021-03-02T12:02:35.824 INFO:teuthology.orchestra.run.smithi036.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-02T12:02:35.825 INFO:teuthology.orchestra.run.smithi036.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-02T12:02:35.825 INFO:teuthology.orchestra.run.smithi036.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-02T12:02:35.825 INFO:teuthology.orchestra.run.smithi036.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-02T12:02:35.825 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi036.front.sepia.ceph.com -o noatime 2021-03-02T12:02:35.826 DEBUG:teuthology.orchestra.run.smithi036:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-03-02T12:02:35.875 DEBUG:teuthology.orchestra.run.smithi036:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-03-02T12:02:35.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-02T12:02:35.889 INFO:teuthology.orchestra.run.smithi036.stderr:sudo: /sbin/restorecon: command not found 2021-03-02T12:02:35.889 DEBUG:teuthology.orchestra.run.smithi036:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:35.989 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:35.986+0000 7f239b1a3d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-03-02T12:02:35.989 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:35.986+0000 7f239b1a3d80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-03-02T12:02:35.990 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:35.986+0000 7f239b1a3d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-02T12:02:35.990 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:35.986+0000 7f239b1a3d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-02T12:02:35.991 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:35.986+0000 7f239b1a3d80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-02T12:02:35.992 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:35.990+0000 7f239b1a3d80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-03-02T12:02:36.017 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:36.014+0000 7f239b1a3d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-02T12:02:37.902 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-03-02T12:02:37.917 DEBUG:teuthology.orchestra.run.smithi036:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:38.003 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:37.998+0000 7f344c1e6d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-03-02T12:02:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:37.998+0000 7f344c1e6d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-03-02T12:02:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:38.002+0000 7f344c1e6d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-02T12:02:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:38.002+0000 7f344c1e6d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-02T12:02:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:38.002+0000 7f344c1e6d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-02T12:02:38.005 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:38.002+0000 7f344c1e6d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-03-02T12:02:38.022 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:38.018+0000 7f344c1e6d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-02T12:02:39.914 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-03-02T12:02:39.976 DEBUG:teuthology.orchestra.run.smithi036:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:40.057 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.054+0000 7f286bf08d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-03-02T12:02:40.058 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.054+0000 7f286bf08d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-03-02T12:02:40.058 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.054+0000 7f286bf08d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-02T12:02:40.059 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.054+0000 7f286bf08d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-02T12:02:40.059 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.054+0000 7f286bf08d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-02T12:02:40.060 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.054+0000 7f286bf08d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-03-02T12:02:40.073 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:02:40.070+0000 7f286bf08d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-02T12:02:41.973 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-03-02T12:02:42.036 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-02T12:02:42.037 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:42.089 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.WV0j4aWlZa 2021-03-02T12:02:42.089 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/mgr/c2-x/keyring /tmp/tmp.WV0j4aWlZa 2021-03-02T12:02:42.147 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.WV0j4aWlZa 2021-03-02T12:02:42.274 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.WV0j4aWlZa is 56B 2021-03-02T12:02:42.663 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.WV0j4aWlZa 2021-03-02T12:02:42.671 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:42.720 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.1EMXBpgA4q 2021-03-02T12:02:42.721 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/osd/c2-0/keyring /tmp/tmp.1EMXBpgA4q 2021-03-02T12:02:42.778 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.1EMXBpgA4q 2021-03-02T12:02:42.904 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.1EMXBpgA4q is 56B 2021-03-02T12:02:42.994 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.1EMXBpgA4q 2021-03-02T12:02:43.003 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:43.053 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.MIdCgVcsYd 2021-03-02T12:02:43.054 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/osd/c2-1/keyring /tmp/tmp.MIdCgVcsYd 2021-03-02T12:02:43.111 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.MIdCgVcsYd 2021-03-02T12:02:43.234 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.MIdCgVcsYd is 56B 2021-03-02T12:02:43.297 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.MIdCgVcsYd 2021-03-02T12:02:43.307 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:43.353 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.4yO2iyeRbt 2021-03-02T12:02:43.354 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/osd/c2-2/keyring /tmp/tmp.4yO2iyeRbt 2021-03-02T12:02:43.410 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.4yO2iyeRbt 2021-03-02T12:02:43.528 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.4yO2iyeRbt is 56B 2021-03-02T12:02:43.814 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.4yO2iyeRbt 2021-03-02T12:02:43.898 DEBUG:teuthology.orchestra.remote:smithi036:/etc/ceph/c2.client.0.keyring is 59B 2021-03-02T12:02:44.522 DEBUG:teuthology.orchestra.remote:smithi036:/etc/ceph/c2.client.1.keyring is 59B 2021-03-02T12:02:44.552 INFO:tasks.ceph:Adding keys to all mons... 2021-03-02T12:02:44.553 DEBUG:teuthology.orchestra.run.smithi036:> sudo tee -a /etc/ceph/c2.keyring 2021-03-02T12:02:44.571 INFO:teuthology.orchestra.run.smithi036.stdout:[mgr.x] 2021-03-02T12:02:44.572 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQBYKT5gDlxnLBAABJi+0D3npG5djRRfE1CLQA== 2021-03-02T12:02:44.572 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.0] 2021-03-02T12:02:44.573 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQBbKT5g8zL3OhAAzkdeeWPl9LjThHXF7pfbeA== 2021-03-02T12:02:44.573 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.1] 2021-03-02T12:02:44.574 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQBeKT5gBHhBABAAzM6glCrFlaoPTFX9t5KhtA== 2021-03-02T12:02:44.574 INFO:teuthology.orchestra.run.smithi036.stdout:[osd.2] 2021-03-02T12:02:44.575 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQBgKT5gC3V2AxAAz3+WLEkdpTB+wcPJlSBv0g== 2021-03-02T12:02:44.575 INFO:teuthology.orchestra.run.smithi036.stdout:[client.0] 2021-03-02T12:02:44.575 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQBZKT5g69PECRAATG81hG4W+jbhxn7MoQ3WcA== 2021-03-02T12:02:44.576 INFO:teuthology.orchestra.run.smithi036.stdout:[client.1] 2021-03-02T12:02:44.577 INFO:teuthology.orchestra.run.smithi036.stdout: key = AQBZKT5gKhfANBAAIUcXo/lGbxqZVbmlCfGMlw== 2021-03-02T12:02:44.578 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-02T12:02:44.659 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-02T12:02:44.750 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-02T12:02:44.844 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-02T12:02:44.929 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-02T12:02:45.038 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-02T12:02:45.143 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-02T12:02:45.144 DEBUG:teuthology.orchestra.run.smithi036:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-03-02T12:02:45.212 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-03-02T12:02:45.319 DEBUG:teuthology.orchestra.run.smithi036:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-03-02T12:02:45.340 DEBUG:teuthology.orchestra.run.smithi036:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-03-02T12:02:45.402 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-03-02T12:02:45.402 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-03-02T12:02:45.403 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-03-02T12:02:45.451 INFO:tasks.ceph.c2.mon.a:Started 2021-03-02T12:02:45.451 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-03-02T12:02:45.452 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-03-02T12:02:45.452 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-03-02T12:02:45.456 INFO:tasks.ceph.c2.mgr.x:Started 2021-03-02T12:02:45.456 INFO:tasks.ceph:Setting crush tunables to default 2021-03-02T12:02:45.457 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c2 osd crush tunables default 2021-03-02T12:02:45.803 INFO:teuthology.orchestra.run.smithi036.stderr:adjusted tunables profile to default 2021-03-02T12:02:45.817 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-03-02T12:02:45.818 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:45.828 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.0NUkUeerwL 2021-03-02T12:02:45.829 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/osd/c2-0/fsid /tmp/tmp.0NUkUeerwL 2021-03-02T12:02:45.888 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.0NUkUeerwL 2021-03-02T12:02:46.003 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.0NUkUeerwL is 37B 2021-03-02T12:02:46.024 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.0NUkUeerwL 2021-03-02T12:02:46.033 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:46.085 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.9A27CFywqk 2021-03-02T12:02:46.086 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/osd/c2-1/fsid /tmp/tmp.9A27CFywqk 2021-03-02T12:02:46.142 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.9A27CFywqk 2021-03-02T12:02:46.251 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.9A27CFywqk is 37B 2021-03-02T12:02:46.266 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.9A27CFywqk 2021-03-02T12:02:46.273 DEBUG:teuthology.orchestra.run.smithi036:> mktemp 2021-03-02T12:02:46.320 INFO:teuthology.orchestra.run.smithi036.stdout:/tmp/tmp.kO0G3986xJ 2021-03-02T12:02:46.321 DEBUG:teuthology.orchestra.run.smithi036:> sudo cp /var/lib/ceph/osd/c2-2/fsid /tmp/tmp.kO0G3986xJ 2021-03-02T12:02:46.385 DEBUG:teuthology.orchestra.run.smithi036:> sudo chmod 0666 /tmp/tmp.kO0G3986xJ 2021-03-02T12:02:46.500 DEBUG:teuthology.orchestra.remote:smithi036:/tmp/tmp.kO0G3986xJ is 37B 2021-03-02T12:02:46.513 DEBUG:teuthology.orchestra.run.smithi036:> rm -fr /tmp/tmp.kO0G3986xJ 2021-03-02T12:02:46.521 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c2 osd new 98f31dad-6607-4f21-aabc-112ed67835de 0 2021-03-02T12:02:46.760 INFO:teuthology.orchestra.run.smithi036.stdout:0 2021-03-02T12:02:46.773 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c2 osd new c849dea4-823d-4929-80b4-b918eb00cd1c 1 2021-03-02T12:02:46.977 INFO:teuthology.orchestra.run.smithi036.stdout:1 2021-03-02T12:02:46.988 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c2 osd new 2cd44c1d-a839-430c-a77c-45e6137327c6 2 2021-03-02T12:02:47.187 INFO:teuthology.orchestra.run.smithi036.stdout:2 2021-03-02T12:02:47.204 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-03-02T12:02:47.205 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-03-02T12:02:47.207 INFO:tasks.ceph.c2.osd.0:Started 2021-03-02T12:02:47.207 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-03-02T12:02:47.208 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-03-02T12:02:47.209 INFO:tasks.ceph.c2.osd.1:Started 2021-03-02T12:02:47.210 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-03-02T12:02:47.210 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-03-02T12:02:47.212 INFO:tasks.ceph.c2.osd.2:Started 2021-03-02T12:02:47.213 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-02T12:02:47.517 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-03-02T12:02:47.694 INFO:teuthology.misc.health.smithi036.stdout: 2021-03-02T12:02:47.695 INFO:teuthology.misc.health.smithi036.stdout:{"epoch":5,"fsid":"c58b7310-abb3-4c11-b73a-9b0fb0709dc8","created":"2021-03-02T12:02:45.598596+0000","modified":"2021-03-02T12:02:47.186009+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"98f31dad-6607-4f21-aabc-112ed67835de","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"c849dea4-823d-4929-80b4-b918eb00cd1c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"2cd44c1d-a839-430c-a77c-45e6137327c6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:47.705 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-03-02T12:02:47.808 INFO:tasks.ceph.c2.osd.2.smithi036.stderr:2021-03-02T12:02:47.806+0000 7f84257fdd80 -1 Falling back to public interface 2021-03-02T12:02:47.825 INFO:tasks.ceph.c2.osd.1.smithi036.stderr:2021-03-02T12:02:47.822+0000 7f1575912d80 -1 Falling back to public interface 2021-03-02T12:02:47.825 INFO:tasks.ceph.c2.osd.0.smithi036.stderr:2021-03-02T12:02:47.822+0000 7f19283d7d80 -1 Falling back to public interface 2021-03-02T12:02:48.425 INFO:tasks.ceph.c2.osd.2.smithi036.stderr:2021-03-02T12:02:48.422+0000 7f84257fdd80 -1 osd.2 0 log_to_monitors {default=true} 2021-03-02T12:02:48.442 INFO:tasks.ceph.c2.osd.1.smithi036.stderr:2021-03-02T12:02:48.438+0000 7f1575912d80 -1 osd.1 0 log_to_monitors {default=true} 2021-03-02T12:02:48.447 INFO:tasks.ceph.c2.osd.0.smithi036.stderr:2021-03-02T12:02:48.446+0000 7f19283d7d80 -1 osd.0 0 log_to_monitors {default=true} 2021-03-02T12:02:49.636 INFO:tasks.ceph.c2.osd.2.smithi036.stderr:2021-03-02T12:02:49.634+0000 7f840f249700 -1 osd.2 0 waiting for initial osdmap 2021-03-02T12:02:49.637 INFO:tasks.ceph.c2.osd.0.smithi036.stderr:2021-03-02T12:02:49.634+0000 7f1911e23700 -1 osd.0 0 waiting for initial osdmap 2021-03-02T12:02:49.637 INFO:tasks.ceph.c2.osd.1.smithi036.stderr:2021-03-02T12:02:49.634+0000 7f155f35e700 -1 osd.1 0 waiting for initial osdmap 2021-03-02T12:02:54.009 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-03-02T12:02:54.283 INFO:teuthology.misc.health.smithi036.stdout: 2021-03-02T12:02:54.657 INFO:teuthology.misc.health.smithi036.stdout:{"epoch":10,"fsid":"c58b7310-abb3-4c11-b73a-9b0fb0709dc8","created":"2021-03-02T12:02:45.598596+0000","modified":"2021-03-02T12:02:52.647508+0000","last_up_change":"2021-03-02T12:02:50.634166+0000","last_in_change":"2021-03-02T12:02:50.634166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:49.454704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"98f31dad-6607-4f21-aabc-112ed67835de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6809","nonce":11767},{"type":"v1","addr":"172.21.15.36:6811","nonce":11767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6813","nonce":11767},{"type":"v1","addr":"172.21.15.36:6815","nonce":11767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6821","nonce":11767},{"type":"v1","addr":"172.21.15.36:6823","nonce":11767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6817","nonce":11767},{"type":"v1","addr":"172.21.15.36:6819","nonce":11767}]},"public_addr":"172.21.15.36:6811/11767","cluster_addr":"172.21.15.36:6815/11767","heartbeat_back_addr":"172.21.15.36:6823/11767","heartbeat_front_addr":"172.21.15.36:6819/11767","state":["exists","up"]},{"osd":1,"uuid":"c849dea4-823d-4929-80b4-b918eb00cd1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":11768},{"type":"v1","addr":"172.21.15.36:6810","nonce":11768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":11768},{"type":"v1","addr":"172.21.15.36:6814","nonce":11768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":11768},{"type":"v1","addr":"172.21.15.36:6822","nonce":11768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":11768},{"type":"v1","addr":"172.21.15.36:6818","nonce":11768}]},"public_addr":"172.21.15.36:6810/11768","cluster_addr":"172.21.15.36:6814/11768","heartbeat_back_addr":"172.21.15.36:6822/11768","heartbeat_front_addr":"172.21.15.36:6818/11768","state":["exists","up"]},{"osd":2,"uuid":"2cd44c1d-a839-430c-a77c-45e6137327c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":11766},{"type":"v1","addr":"172.21.15.36:6801","nonce":11766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":11766},{"type":"v1","addr":"172.21.15.36:6803","nonce":11766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":11766},{"type":"v1","addr":"172.21.15.36:6807","nonce":11766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":11766},{"type":"v1","addr":"172.21.15.36:6805","nonce":11766}]},"public_addr":"172.21.15.36:6801/11766","cluster_addr":"172.21.15.36:6803/11766","heartbeat_back_addr":"172.21.15.36:6807/11766","heartbeat_front_addr":"172.21.15.36:6805/11766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.456667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.396713+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.470707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:54.658 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-03-02T12:02:54.659 INFO:tasks.ceph:Creating RBD pool 2021-03-02T12:02:54.659 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c2 osd pool create rbd 8 2021-03-02T12:02:55.676 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'rbd' created 2021-03-02T12:02:55.692 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-03-02T12:02:56.686 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rbd' on pool 'rbd' 2021-03-02T12:02:56.701 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-03-02T12:02:56.702 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-03-02T12:02:56.702 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-03-02T12:02:56.987 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:56.988 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":12,"fsid":"c58b7310-abb3-4c11-b73a-9b0fb0709dc8","created":"2021-03-02T12:02:45.598596+0000","modified":"2021-03-02T12:02:56.668745+0000","last_up_change":"2021-03-02T12:02:50.634166+0000","last_in_change":"2021-03-02T12:02:50.634166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:49.454704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-02T12:02:54.944487+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"98f31dad-6607-4f21-aabc-112ed67835de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6809","nonce":11767},{"type":"v1","addr":"172.21.15.36:6811","nonce":11767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6813","nonce":11767},{"type":"v1","addr":"172.21.15.36:6815","nonce":11767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6821","nonce":11767},{"type":"v1","addr":"172.21.15.36:6823","nonce":11767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6817","nonce":11767},{"type":"v1","addr":"172.21.15.36:6819","nonce":11767}]},"public_addr":"172.21.15.36:6811/11767","cluster_addr":"172.21.15.36:6815/11767","heartbeat_back_addr":"172.21.15.36:6823/11767","heartbeat_front_addr":"172.21.15.36:6819/11767","state":["exists","up"]},{"osd":1,"uuid":"c849dea4-823d-4929-80b4-b918eb00cd1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":11768},{"type":"v1","addr":"172.21.15.36:6810","nonce":11768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":11768},{"type":"v1","addr":"172.21.15.36:6814","nonce":11768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":11768},{"type":"v1","addr":"172.21.15.36:6822","nonce":11768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":11768},{"type":"v1","addr":"172.21.15.36:6818","nonce":11768}]},"public_addr":"172.21.15.36:6810/11768","cluster_addr":"172.21.15.36:6814/11768","heartbeat_back_addr":"172.21.15.36:6822/11768","heartbeat_front_addr":"172.21.15.36:6818/11768","state":["exists","up"]},{"osd":2,"uuid":"2cd44c1d-a839-430c-a77c-45e6137327c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":11766},{"type":"v1","addr":"172.21.15.36:6801","nonce":11766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":11766},{"type":"v1","addr":"172.21.15.36:6803","nonce":11766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":11766},{"type":"v1","addr":"172.21.15.36:6807","nonce":11766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":11766},{"type":"v1","addr":"172.21.15.36:6805","nonce":11766}]},"public_addr":"172.21.15.36:6801/11766","cluster_addr":"172.21.15.36:6803/11766","heartbeat_back_addr":"172.21.15.36:6807/11766","heartbeat_front_addr":"172.21.15.36:6805/11766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.456667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.396713+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.470707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:57.008 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-02T12:02:49.454704+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-03-02T12:02:54.944487+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-03-02T12:02:57.008 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get device_health_metrics pg_num 2021-03-02T12:02:57.283 INFO:teuthology.orchestra.run.smithi036.stdout:pg_num: 1 2021-03-02T12:02:57.296 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd pool get rbd pg_num 2021-03-02T12:02:57.572 INFO:teuthology.orchestra.run.smithi036.stdout:pg_num: 8 2021-03-02T12:02:57.585 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-03-02T12:02:57.586 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-03-02T12:02:57.586 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early mgr dump --format=json 2021-03-02T12:02:57.890 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:57.903 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6824","nonce":11557},{"type":"v1","addr":"172.21.15.36:6825","nonce":11557}]},"active_addr":"172.21.15.36:6825/11557","active_change":"2021-03-02T12:02:49.422759+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.6.2","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.36:0","nonce":2848998679}]},{"addrvec":[{"type":"v2","addr":"172.21.15.36:0","nonce":2511813070}]},{"addrvec":[{"type":"v2","addr":"172.21.15.36:0","nonce":2040182572}]}]}} 2021-03-02T12:02:57.905 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-03-02T12:02:57.906 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-03-02T12:02:57.906 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-03-02T12:02:58.180 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:58.181 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":13,"fsid":"c58b7310-abb3-4c11-b73a-9b0fb0709dc8","created":"2021-03-02T12:02:45.598596+0000","modified":"2021-03-02T12:02:57.675269+0000","last_up_change":"2021-03-02T12:02:50.634166+0000","last_in_change":"2021-03-02T12:02:50.634166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:49.454704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-02T12:02:54.944487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"98f31dad-6607-4f21-aabc-112ed67835de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6809","nonce":11767},{"type":"v1","addr":"172.21.15.36:6811","nonce":11767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6813","nonce":11767},{"type":"v1","addr":"172.21.15.36:6815","nonce":11767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6821","nonce":11767},{"type":"v1","addr":"172.21.15.36:6823","nonce":11767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6817","nonce":11767},{"type":"v1","addr":"172.21.15.36:6819","nonce":11767}]},"public_addr":"172.21.15.36:6811/11767","cluster_addr":"172.21.15.36:6815/11767","heartbeat_back_addr":"172.21.15.36:6823/11767","heartbeat_front_addr":"172.21.15.36:6819/11767","state":["exists","up"]},{"osd":1,"uuid":"c849dea4-823d-4929-80b4-b918eb00cd1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":11768},{"type":"v1","addr":"172.21.15.36:6810","nonce":11768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":11768},{"type":"v1","addr":"172.21.15.36:6814","nonce":11768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":11768},{"type":"v1","addr":"172.21.15.36:6822","nonce":11768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":11768},{"type":"v1","addr":"172.21.15.36:6818","nonce":11768}]},"public_addr":"172.21.15.36:6810/11768","cluster_addr":"172.21.15.36:6814/11768","heartbeat_back_addr":"172.21.15.36:6822/11768","heartbeat_front_addr":"172.21.15.36:6818/11768","state":["exists","up"]},{"osd":2,"uuid":"2cd44c1d-a839-430c-a77c-45e6137327c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":11766},{"type":"v1","addr":"172.21.15.36:6801","nonce":11766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":11766},{"type":"v1","addr":"172.21.15.36:6803","nonce":11766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":11766},{"type":"v1","addr":"172.21.15.36:6807","nonce":11766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":11766},{"type":"v1","addr":"172.21.15.36:6805","nonce":11766}]},"public_addr":"172.21.15.36:6801/11766","cluster_addr":"172.21.15.36:6803/11766","heartbeat_back_addr":"172.21.15.36:6807/11766","heartbeat_front_addr":"172.21.15.36:6805/11766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.456667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.396713+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.470707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:58.194 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-03-02T12:02:58.195 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd dump --format=json 2021-03-02T12:02:58.472 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:02:58.473 INFO:teuthology.orchestra.run.smithi036.stdout:{"epoch":13,"fsid":"c58b7310-abb3-4c11-b73a-9b0fb0709dc8","created":"2021-03-02T12:02:45.598596+0000","modified":"2021-03-02T12:02:57.675269+0000","last_up_change":"2021-03-02T12:02:50.634166+0000","last_in_change":"2021-03-02T12:02:50.634166+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-02T12:02:49.454704+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-02T12:02:54.944487+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"98f31dad-6607-4f21-aabc-112ed67835de","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6809","nonce":11767},{"type":"v1","addr":"172.21.15.36:6811","nonce":11767}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6813","nonce":11767},{"type":"v1","addr":"172.21.15.36:6815","nonce":11767}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6821","nonce":11767},{"type":"v1","addr":"172.21.15.36:6823","nonce":11767}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6817","nonce":11767},{"type":"v1","addr":"172.21.15.36:6819","nonce":11767}]},"public_addr":"172.21.15.36:6811/11767","cluster_addr":"172.21.15.36:6815/11767","heartbeat_back_addr":"172.21.15.36:6823/11767","heartbeat_front_addr":"172.21.15.36:6819/11767","state":["exists","up"]},{"osd":1,"uuid":"c849dea4-823d-4929-80b4-b918eb00cd1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6808","nonce":11768},{"type":"v1","addr":"172.21.15.36:6810","nonce":11768}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6812","nonce":11768},{"type":"v1","addr":"172.21.15.36:6814","nonce":11768}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6820","nonce":11768},{"type":"v1","addr":"172.21.15.36:6822","nonce":11768}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6816","nonce":11768},{"type":"v1","addr":"172.21.15.36:6818","nonce":11768}]},"public_addr":"172.21.15.36:6810/11768","cluster_addr":"172.21.15.36:6814/11768","heartbeat_back_addr":"172.21.15.36:6822/11768","heartbeat_front_addr":"172.21.15.36:6818/11768","state":["exists","up"]},{"osd":2,"uuid":"2cd44c1d-a839-430c-a77c-45e6137327c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6800","nonce":11766},{"type":"v1","addr":"172.21.15.36:6801","nonce":11766}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6802","nonce":11766},{"type":"v1","addr":"172.21.15.36:6803","nonce":11766}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6806","nonce":11766},{"type":"v1","addr":"172.21.15.36:6807","nonce":11766}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.36:6804","nonce":11766},{"type":"v1","addr":"172.21.15.36:6805","nonce":11766}]},"public_addr":"172.21.15.36:6801/11766","cluster_addr":"172.21.15.36:6803/11766","heartbeat_back_addr":"172.21.15.36:6807/11766","heartbeat_front_addr":"172.21.15.36:6805/11766","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.456667+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.396713+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-02T12:02:49.470707+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-02T12:02:58.483 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.0 flush_pg_stats 2021-03-02T12:02:58.649 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2021-03-02T12:02:58.650 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.1 flush_pg_stats 2021-03-02T12:02:58.812 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2021-03-02T12:02:58.813 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early tell osd.2 flush_pg_stats 2021-03-02T12:02:58.974 INFO:teuthology.orchestra.run.smithi036.stdout:34359738372 2021-03-02T12:02:58.975 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2021-03-02T12:02:59.251 INFO:teuthology.orchestra.run.smithi036.stdout:34359738370 2021-03-02T12:02:59.270 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738370 for osd.0 2021-03-02T12:03:00.272 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.0 2021-03-02T12:03:00.554 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2021-03-02T12:03:00.567 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.0 2021-03-02T12:03:00.568 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.1 2021-03-02T12:03:00.849 INFO:teuthology.orchestra.run.smithi036.stdout:34359738371 2021-03-02T12:03:00.863 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2021-03-02T12:03:00.864 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early osd last-stat-seq osd.2 2021-03-02T12:03:01.136 INFO:teuthology.orchestra.run.smithi036.stdout:34359738372 2021-03-02T12:03:01.150 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738372 got 34359738372 for osd.2 2021-03-02T12:03:01.151 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-03-02T12:03:01.151 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2021-03-02T12:03:01.419 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:03:01.732 INFO:teuthology.orchestra.run.smithi036.stderr:dumped all 2021-03-02T12:03:01.732 INFO:teuthology.orchestra.run.smithi036.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-03-02T12:02:59.433553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.758071"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699085+0000","last_change":"2021-03-02T12:02:56.695198+0000","last_active":"2021-03-02T12:02:56.699085+0000","last_peered":"2021-03-02T12:02:56.699085+0000","last_clean":"2021-03-02T12:02:56.699085+0000","last_became_active":"2021-03-02T12:02:56.694167+0000","last_became_peered":"2021-03-02T12:02:56.694167+0000","last_unstale":"2021-03-02T12:02:56.699085+0000","last_undegraded":"2021-03-02T12:02:56.699085+0000","last_fullsized":"2021-03-02T12:02:56.699085+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699116+0000","last_change":"2021-03-02T12:02:56.695548+0000","last_active":"2021-03-02T12:02:56.699116+0000","last_peered":"2021-03-02T12:02:56.699116+0000","last_clean":"2021-03-02T12:02:56.699116+0000","last_became_active":"2021-03-02T12:02:56.693876+0000","last_became_peered":"2021-03-02T12:02:56.693876+0000","last_unstale":"2021-03-02T12:02:56.699116+0000","last_undegraded":"2021-03-02T12:02:56.699116+0000","last_fullsized":"2021-03-02T12:02:56.699116+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.698543+0000","last_change":"2021-03-02T12:02:56.694372+0000","last_active":"2021-03-02T12:02:56.698543+0000","last_peered":"2021-03-02T12:02:56.698543+0000","last_clean":"2021-03-02T12:02:56.698543+0000","last_became_active":"2021-03-02T12:02:56.693861+0000","last_became_peered":"2021-03-02T12:02:56.693861+0000","last_unstale":"2021-03-02T12:02:56.698543+0000","last_undegraded":"2021-03-02T12:02:56.698543+0000","last_fullsized":"2021-03-02T12:02:56.698543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.693113+0000","last_change":"2021-03-02T12:02:56.689093+0000","last_active":"2021-03-02T12:02:56.693113+0000","last_peered":"2021-03-02T12:02:56.693113+0000","last_clean":"2021-03-02T12:02:56.693113+0000","last_became_active":"2021-03-02T12:02:56.688648+0000","last_became_peered":"2021-03-02T12:02:56.688648+0000","last_unstale":"2021-03-02T12:02:56.693113+0000","last_undegraded":"2021-03-02T12:02:56.693113+0000","last_fullsized":"2021-03-02T12:02:56.693113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.684769+0000","last_change":"2021-03-02T12:02:51.952260+0000","last_active":"2021-03-02T12:02:56.684769+0000","last_peered":"2021-03-02T12:02:56.684769+0000","last_clean":"2021-03-02T12:02:56.684769+0000","last_became_active":"2021-03-02T12:02:51.950643+0000","last_became_peered":"2021-03-02T12:02:51.950643+0000","last_unstale":"2021-03-02T12:02:56.684769+0000","last_undegraded":"2021-03-02T12:02:56.684769+0000","last_fullsized":"2021-03-02T12:02:56.684769+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:49.622861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:49.622861+0000","last_clean_scrub_stamp":"2021-03-02T12:02:49.622861+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.695468+0000","last_change":"2021-03-02T12:02:56.689940+0000","last_active":"2021-03-02T12:02:56.695468+0000","last_peered":"2021-03-02T12:02:56.695468+0000","last_clean":"2021-03-02T12:02:56.695468+0000","last_became_active":"2021-03-02T12:02:56.689493+0000","last_became_peered":"2021-03-02T12:02:56.689493+0000","last_unstale":"2021-03-02T12:02:56.695468+0000","last_undegraded":"2021-03-02T12:02:56.695468+0000","last_fullsized":"2021-03-02T12:02:56.695468+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.693473+0000","last_change":"2021-03-02T12:02:56.689625+0000","last_active":"2021-03-02T12:02:56.693473+0000","last_peered":"2021-03-02T12:02:56.693473+0000","last_clean":"2021-03-02T12:02:56.693473+0000","last_became_active":"2021-03-02T12:02:56.688964+0000","last_became_peered":"2021-03-02T12:02:56.688964+0000","last_unstale":"2021-03-02T12:02:56.693473+0000","last_undegraded":"2021-03-02T12:02:56.693473+0000","last_fullsized":"2021-03-02T12:02:56.693473+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699156+0000","last_change":"2021-03-02T12:02:56.693977+0000","last_active":"2021-03-02T12:02:56.699156+0000","last_peered":"2021-03-02T12:02:56.699156+0000","last_clean":"2021-03-02T12:02:56.699156+0000","last_became_active":"2021-03-02T12:02:56.693514+0000","last_became_peered":"2021-03-02T12:02:56.693514+0000","last_unstale":"2021-03-02T12:02:56.699156+0000","last_undegraded":"2021-03-02T12:02:56.699156+0000","last_fullsized":"2021-03-02T12:02:56.699156+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699097+0000","last_change":"2021-03-02T12:02:56.695098+0000","last_active":"2021-03-02T12:02:56.699097+0000","last_peered":"2021-03-02T12:02:56.699097+0000","last_clean":"2021-03-02T12:02:56.699097+0000","last_became_active":"2021-03-02T12:02:56.694391+0000","last_became_peered":"2021-03-02T12:02:56.694391+0000","last_unstale":"2021-03-02T12:02:56.699097+0000","last_undegraded":"2021-03-02T12:02:56.699097+0000","last_fullsized":"2021-03-02T12:02:56.699097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}]}} 2021-03-02T12:03:01.734 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early pg dump --format=json 2021-03-02T12:03:02.014 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:03:02.244 INFO:teuthology.orchestra.run.smithi036.stderr:dumped all 2021-03-02T12:03:02.245 INFO:teuthology.orchestra.run.smithi036.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-02T12:03:01.434106+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146088,"kb_used_data":336,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426712,"statfs":{"total":322122547200,"available":318900953088,"internally_reserved":3221225472,"allocated":344064,"data_stored":99408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.758624"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699085+0000","last_change":"2021-03-02T12:02:56.695198+0000","last_active":"2021-03-02T12:02:56.699085+0000","last_peered":"2021-03-02T12:02:56.699085+0000","last_clean":"2021-03-02T12:02:56.699085+0000","last_became_active":"2021-03-02T12:02:56.694167+0000","last_became_peered":"2021-03-02T12:02:56.694167+0000","last_unstale":"2021-03-02T12:02:56.699085+0000","last_undegraded":"2021-03-02T12:02:56.699085+0000","last_fullsized":"2021-03-02T12:02:56.699085+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699116+0000","last_change":"2021-03-02T12:02:56.695548+0000","last_active":"2021-03-02T12:02:56.699116+0000","last_peered":"2021-03-02T12:02:56.699116+0000","last_clean":"2021-03-02T12:02:56.699116+0000","last_became_active":"2021-03-02T12:02:56.693876+0000","last_became_peered":"2021-03-02T12:02:56.693876+0000","last_unstale":"2021-03-02T12:02:56.699116+0000","last_undegraded":"2021-03-02T12:02:56.699116+0000","last_fullsized":"2021-03-02T12:02:56.699116+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.698543+0000","last_change":"2021-03-02T12:02:56.694372+0000","last_active":"2021-03-02T12:02:56.698543+0000","last_peered":"2021-03-02T12:02:56.698543+0000","last_clean":"2021-03-02T12:02:56.698543+0000","last_became_active":"2021-03-02T12:02:56.693861+0000","last_became_peered":"2021-03-02T12:02:56.693861+0000","last_unstale":"2021-03-02T12:02:56.698543+0000","last_undegraded":"2021-03-02T12:02:56.698543+0000","last_fullsized":"2021-03-02T12:02:56.698543+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.693113+0000","last_change":"2021-03-02T12:02:56.689093+0000","last_active":"2021-03-02T12:02:56.693113+0000","last_peered":"2021-03-02T12:02:56.693113+0000","last_clean":"2021-03-02T12:02:56.693113+0000","last_became_active":"2021-03-02T12:02:56.688648+0000","last_became_peered":"2021-03-02T12:02:56.688648+0000","last_unstale":"2021-03-02T12:02:56.693113+0000","last_undegraded":"2021-03-02T12:02:56.693113+0000","last_fullsized":"2021-03-02T12:02:56.693113+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.684769+0000","last_change":"2021-03-02T12:02:51.952260+0000","last_active":"2021-03-02T12:02:56.684769+0000","last_peered":"2021-03-02T12:02:56.684769+0000","last_clean":"2021-03-02T12:02:56.684769+0000","last_became_active":"2021-03-02T12:02:51.950643+0000","last_became_peered":"2021-03-02T12:02:51.950643+0000","last_unstale":"2021-03-02T12:02:56.684769+0000","last_undegraded":"2021-03-02T12:02:56.684769+0000","last_fullsized":"2021-03-02T12:02:56.684769+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:49.622861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:49.622861+0000","last_clean_scrub_stamp":"2021-03-02T12:02:49.622861+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.695468+0000","last_change":"2021-03-02T12:02:56.689940+0000","last_active":"2021-03-02T12:02:56.695468+0000","last_peered":"2021-03-02T12:02:56.695468+0000","last_clean":"2021-03-02T12:02:56.695468+0000","last_became_active":"2021-03-02T12:02:56.689493+0000","last_became_peered":"2021-03-02T12:02:56.689493+0000","last_unstale":"2021-03-02T12:02:56.695468+0000","last_undegraded":"2021-03-02T12:02:56.695468+0000","last_fullsized":"2021-03-02T12:02:56.695468+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.693473+0000","last_change":"2021-03-02T12:02:56.689625+0000","last_active":"2021-03-02T12:02:56.693473+0000","last_peered":"2021-03-02T12:02:56.693473+0000","last_clean":"2021-03-02T12:02:56.693473+0000","last_became_active":"2021-03-02T12:02:56.688964+0000","last_became_peered":"2021-03-02T12:02:56.688964+0000","last_unstale":"2021-03-02T12:02:56.693473+0000","last_undegraded":"2021-03-02T12:02:56.693473+0000","last_fullsized":"2021-03-02T12:02:56.693473+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699156+0000","last_change":"2021-03-02T12:02:56.693977+0000","last_active":"2021-03-02T12:02:56.699156+0000","last_peered":"2021-03-02T12:02:56.699156+0000","last_clean":"2021-03-02T12:02:56.699156+0000","last_became_active":"2021-03-02T12:02:56.693514+0000","last_became_peered":"2021-03-02T12:02:56.693514+0000","last_unstale":"2021-03-02T12:02:56.699156+0000","last_undegraded":"2021-03-02T12:02:56.699156+0000","last_fullsized":"2021-03-02T12:02:56.699156+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-02T12:02:56.699097+0000","last_change":"2021-03-02T12:02:56.695098+0000","last_active":"2021-03-02T12:02:56.699097+0000","last_peered":"2021-03-02T12:02:56.699097+0000","last_clean":"2021-03-02T12:02:56.699097+0000","last_became_active":"2021-03-02T12:02:56.694391+0000","last_became_peered":"2021-03-02T12:02:56.694391+0000","last_unstale":"2021-03-02T12:02:56.699097+0000","last_undegraded":"2021-03-02T12:02:56.699097+0000","last_fullsized":"2021-03-02T12:02:56.699097+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-02T12:02:55.662330+0000","last_clean_scrub_stamp":"2021-03-02T12:02:55.662330+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]}]}]}} 2021-03-02T12:03:02.247 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-03-02T12:03:02.247 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-03-02T12:03:02.248 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-03-02T12:03:02.249 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 --log-early health --format=json 2021-03-02T12:03:02.580 INFO:teuthology.orchestra.run.smithi036.stdout: 2021-03-02T12:03:03.325 INFO:teuthology.orchestra.run.smithi036.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-02T12:03:03.326 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-03-02T12:03:03.327 INFO:teuthology.run_tasks:Running task rgw... 2021-03-02T12:03:03.355 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c1.client.1': {'port': 8001}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-03-02T12:03:03.355 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c1.client.1', 'c2.client.0', 'c2.client.1']) 2021-03-02T12:03:03.356 INFO:tasks.rgw:Creating data pools 2021-03-02T12:03:03.356 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-03-02T12:03:03.356 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-03-02T12:03:04.530 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.data' created 2021-03-02T12:03:04.548 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-03-02T12:03:05.542 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-02T12:03:05.560 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-03-02T12:03:06.561 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.index' created 2021-03-02T12:03:06.573 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-03-02T12:03:07.576 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-02T12:03:07.597 DEBUG:tasks.rgw:Obtaining remote for client c1.client.1 2021-03-02T12:03:07.598 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-03-02T12:03:07.887 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.data' already exists 2021-03-02T12:03:07.898 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-03-02T12:03:08.572 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-02T12:03:08.585 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-03-02T12:03:08.870 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'default.rgw.buckets.index' already exists 2021-03-02T12:03:08.886 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-03-02T12:03:09.591 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-02T12:03:09.602 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-03-02T12:03:09.602 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-03-02T12:03:10.592 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.data' created 2021-03-02T12:03:10.607 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-03-02T12:03:11.607 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-02T12:03:11.636 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-03-02T12:03:12.616 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.index' created 2021-03-02T12:03:12.627 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-03-02T12:03:13.640 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-02T12:03:13.662 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-03-02T12:03:13.663 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-03-02T12:03:13.932 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.data' already exists 2021-03-02T12:03:14.038 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-03-02T12:03:14.625 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-02T12:03:14.637 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-03-02T12:03:14.918 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'default.rgw.buckets.index' already exists 2021-03-02T12:03:14.929 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-03-02T12:03:15.635 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-02T12:03:15.647 DEBUG:tasks.rgw:Pools created 2021-03-02T12:03:15.647 INFO:tasks.rgw:Configuring compression type = random 2021-03-02T12:03:15.648 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-03-02T12:03:15.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-03-02T12:03:15.649 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-03-02T12:03:15.687 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:16.032 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:16.595 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:15.697+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.580 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.568+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.581 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.568+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.582 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.572+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.583 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.572+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.584 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.572+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.586 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.572+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.588 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.572+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.594 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.595 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.595 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.595 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 10 cannot find current period zonegroup using local zonegroup 2021-03-02T12:03:17.595 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.596 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.596 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.596 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.596 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 10 Creating default zonegroup 2021-03-02T12:03:17.596 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 20 get_system_obj_state: rctx=0x7ffcaeaf6370 obj=:.pools.avail state=0x562d8927f220 s->prefetch_data=0 2021-03-02T12:03:17.597 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.588+0000 7efd6e0ce080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-03-02T12:03:17.606 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.600+0000 7efd6e0ce080 10 WARNING: store->list_zones() returned r=-2 2021-03-02T12:03:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.600+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.600+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.610 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.604+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.604+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.604+0000 7efd6e0ce080 10 could not read realm id: (2) No such file or directory 2021-03-02T12:03:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.604+0000 7efd6e0ce080 10 WARNING: failed to set zone as default, r=-22 2021-03-02T12:03:17.611 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.604+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.612 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.604+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.623 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.623 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados_obj.operate() r=0 bl.length=839 2021-03-02T12:03:17.623 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.624 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:17.624 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 zonegroup default 2021-03-02T12:03:17.624 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.624 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:17.625 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 10 Cannot find current period zone using local zone 2021-03-02T12:03:17.625 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 10 Using default name default 2021-03-02T12:03:17.625 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.625 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:17.625 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.612+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.616+0000 7efd6e0ce080 20 rados_obj.operate() r=0 bl.length=839 2021-03-02T12:03:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.616+0000 7efd6e0ce080 20 zone default found 2021-03-02T12:03:17.626 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:17.616+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:19.587+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:19.591 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:19.587+0000 7efd6e0ce080 20 rados->read ofs=0 len=0 2021-03-02T12:03:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:19.591+0000 7efd6e0ce080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:19.591+0000 7efd6e0ce080 20 started sync module instance, tier type = 2021-03-02T12:03:19.592 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:19.591+0000 7efd6e0ce080 20 started zone id=edeb7726-8674-41bf-94e4-977bf92f7813 (name=default) with tier type = 2021-03-02T12:03:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=0 2021-03-02T12:03:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=1 2021-03-02T12:03:21.632 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=2 2021-03-02T12:03:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=3 2021-03-02T12:03:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=4 2021-03-02T12:03:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=5 2021-03-02T12:03:21.633 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.626+0000 7efd6e0ce080 20 add_watcher() i=6 2021-03-02T12:03:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.630+0000 7efd6e0ce080 20 add_watcher() i=7 2021-03-02T12:03:21.634 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.630+0000 7efd6e0ce080 2 all 8 watchers are set, enabling cache 2021-03-02T12:03:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.630+0000 7efd6e0ce080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.630+0000 7efd6e0ce080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:21.635 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:21.630+0000 7efd29feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:25.606 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.602+0000 7efd6e0ce080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-03-02T12:03:25.709 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.702+0000 7efc8ffff700 20 reqs_thread_entry: start 2021-03-02T12:03:25.709 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.702+0000 7efc8dffb700 20 reqs_thread_entry: start 2021-03-02T12:03:25.709 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.702+0000 7efd6e0ce080 20 init_complete bucket index max shards: 11 2021-03-02T12:03:25.710 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.702+0000 7efc7bfff700 20 reqs_thread_entry: start 2021-03-02T12:03:25.725 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.722+0000 7efd29feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:25.727 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.722+0000 7efd6e0ce080 20 remove_watcher() i=0 2021-03-02T12:03:25.728 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.722+0000 7efd6e0ce080 2 removed watcher, disabling cache 2021-03-02T12:03:25.732 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.726+0000 7efd6e0ce080 20 remove_watcher() i=1 2021-03-02T12:03:25.734 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.730+0000 7efd6e0ce080 20 remove_watcher() i=2 2021-03-02T12:03:25.737 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.734+0000 7efd6e0ce080 20 remove_watcher() i=3 2021-03-02T12:03:25.739 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.734+0000 7efd6e0ce080 20 remove_watcher() i=4 2021-03-02T12:03:25.744 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.738+0000 7efd6e0ce080 20 remove_watcher() i=5 2021-03-02T12:03:25.746 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.742+0000 7efd6e0ce080 20 remove_watcher() i=6 2021-03-02T12:03:25.748 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.742+0000 7efd6e0ce080 20 remove_watcher() i=7 2021-03-02T12:03:25.757 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:03:25.758 DEBUG:tasks.util.rgw: json result: [] 2021-03-02T12:03:25.758 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:25.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:25.759 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-02T12:03:25.842 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:25.843 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.850+0000 7fd3fe36e080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:25.853 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.850+0000 7fd3fe36e080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:25.861 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.858+0000 7fd332ffd700 20 reqs_thread_entry: start 2021-03-02T12:03:25.861 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.858+0000 7fd3fe36e080 20 rados->read ofs=0 len=0 2021-03-02T12:03:25.863 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.858+0000 7fd3fe36e080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:25.863 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.858+0000 7fd3fe36e080 20 rados->read ofs=0 len=0 2021-03-02T12:03:25.864 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.858+0000 7fd3fe36e080 20 rados_obj.operate() r=0 bl.length=839 2021-03-02T12:03:25.864 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.858+0000 7fd3fe36e080 20 rados->read ofs=0 len=0 2021-03-02T12:03:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.862+0000 7fd3fe36e080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:25.865 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.862+0000 7fd3fe36e080 20 rados->read ofs=0 len=0 2021-03-02T12:03:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.862+0000 7fd3fe36e080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:25.866 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.862+0000 7fd3fe36e080 20 rados->read ofs=0 len=0 2021-03-02T12:03:25.867 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:25.862+0000 7fd3fe36e080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:25.881 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"edeb7726-8674-41bf-94e4-977bf92f7813","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:03:25.881 DEBUG:tasks.util.rgw: json result: {'id': 'edeb7726-8674-41bf-94e4-977bf92f7813', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-02T12:03:25.882 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['user', 'list'] 2021-03-02T12:03:25.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'user', 'list'] 2021-03-02T12:03:25.883 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 user list 2021-03-02T12:03:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:26.371 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:26.380 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.374+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.378+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.383 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.378+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.385 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.378+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.386 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.378+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.387 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.378+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.389 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.378+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.390 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.382+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:26.405 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 RGWRados::pool_iterate: got zonegroup_info.63ca3227-10da-47d3-b7cb-c4507fd6de77 2021-03-02T12:03:26.406 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 RGWRados::pool_iterate: got zone_info.edeb7726-8674-41bf-94e4-977bf92f7813 2021-03-02T12:03:26.406 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 RGWRados::pool_iterate: got zone_names.default 2021-03-02T12:03:26.407 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-03-02T12:03:26.407 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:26.408 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.415 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=855 2021-03-02T12:03:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:26.416 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.402+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.417 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 10 cannot find current period zonegroup using local zonegroup 2021-03-02T12:03:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.418 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.419 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.419 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:26.419 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.419 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:26.420 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 zonegroup default 2021-03-02T12:03:26.420 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.420 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.421 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 10 Cannot find current period zone using local zone 2021-03-02T12:03:26.421 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.406+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.421 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.422 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.422 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:26.422 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados_obj.operate() r=0 bl.length=855 2021-03-02T12:03:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 zone default found 2021-03-02T12:03:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.423 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados->read ofs=0 len=0 2021-03-02T12:03:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:26.424 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 started sync module instance, tier type = 2021-03-02T12:03:26.425 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.410+0000 7fedb59ac080 20 started zone id=edeb7726-8674-41bf-94e4-977bf92f7813 (name=default) with tier type = 2021-03-02T12:03:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.434+0000 7fedb59ac080 20 add_watcher() i=0 2021-03-02T12:03:26.440 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.434+0000 7fedb59ac080 20 add_watcher() i=1 2021-03-02T12:03:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.434+0000 7fedb59ac080 20 add_watcher() i=2 2021-03-02T12:03:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.434+0000 7fedb59ac080 20 add_watcher() i=3 2021-03-02T12:03:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.434+0000 7fedb59ac080 20 add_watcher() i=4 2021-03-02T12:03:26.441 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.434+0000 7fedb59ac080 20 add_watcher() i=5 2021-03-02T12:03:26.442 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.438+0000 7fedb59ac080 20 add_watcher() i=6 2021-03-02T12:03:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.438+0000 7fedb59ac080 20 add_watcher() i=7 2021-03-02T12:03:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.438+0000 7fedb59ac080 2 all 8 watchers are set, enabling cache 2021-03-02T12:03:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.438+0000 7fedb59ac080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.438+0000 7fedb59ac080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:26.443 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.438+0000 7fed72fdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:26.530 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.526+0000 7fece17fa700 20 reqs_thread_entry: start 2021-03-02T12:03:26.916 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.526+0000 7fedb59ac080 20 init_complete bucket index max shards: 11 2021-03-02T12:03:26.917 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.526+0000 7fecdaffd700 20 reqs_thread_entry: start 2021-03-02T12:03:26.918 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.526+0000 7fecd97fa700 20 reqs_thread_entry: start 2021-03-02T12:03:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.546+0000 7fed72fdd700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.550+0000 7fedb59ac080 20 remove_watcher() i=0 2021-03-02T12:03:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.550+0000 7fedb59ac080 2 removed watcher, disabling cache 2021-03-02T12:03:26.921 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.554+0000 7fedb59ac080 20 remove_watcher() i=1 2021-03-02T12:03:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.554+0000 7fedb59ac080 20 remove_watcher() i=2 2021-03-02T12:03:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.558+0000 7fedb59ac080 20 remove_watcher() i=3 2021-03-02T12:03:26.922 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.562+0000 7fedb59ac080 20 remove_watcher() i=4 2021-03-02T12:03:26.934 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.562+0000 7fedb59ac080 20 remove_watcher() i=5 2021-03-02T12:03:26.935 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.566+0000 7fedb59ac080 20 remove_watcher() i=6 2021-03-02T12:03:26.935 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:26.566+0000 7fedb59ac080 20 remove_watcher() i=7 2021-03-02T12:03:26.935 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:03:26.936 DEBUG:tasks.util.rgw: json result: [] 2021-03-02T12:03:26.936 INFO:tasks.util.rgw:rgwadmin: c1.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:26.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:26.937 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-02T12:03:27.006 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:27.006 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:27.018 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.014+0000 7f52bc44f080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:27.018 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.014+0000 7f52bc44f080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.022+0000 7f51eb7fe700 20 reqs_thread_entry: start 2021-03-02T12:03:27.027 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.022+0000 7f52bc44f080 20 rados->read ofs=0 len=0 2021-03-02T12:03:27.028 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.022+0000 7f52bc44f080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:27.029 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.022+0000 7f52bc44f080 20 rados->read ofs=0 len=0 2021-03-02T12:03:27.030 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados_obj.operate() r=0 bl.length=855 2021-03-02T12:03:27.031 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados->read ofs=0 len=0 2021-03-02T12:03:27.031 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados->read ofs=0 len=0 2021-03-02T12:03:27.032 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:27.033 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados->read ofs=0 len=0 2021-03-02T12:03:27.034 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:27.026+0000 7f52bc44f080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:27.041 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"edeb7726-8674-41bf-94e4-977bf92f7813","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:03:27.041 DEBUG:tasks.util.rgw: json result: {'id': 'edeb7726-8674-41bf-94e4-977bf92f7813', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-02T12:03:27.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-03-02T12:03:27.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-03-02T12:03:27.043 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-03-02T12:03:27.080 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:27.081 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:27.091 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:27.085+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.605 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.601+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.605 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.601+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.610 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.605+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.610 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.605+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.611 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.609+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.611 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.609+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.615 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.609+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.631 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.631 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.631 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.631 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.632 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.632 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.632 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 10 cannot find current period zonegroup using local zonegroup 2021-03-02T12:03:28.632 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.625+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.633 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.629+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.634 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.629+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.634 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.629+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.634 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.629+0000 7f8fe661c080 10 Creating default zonegroup 2021-03-02T12:03:28.634 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.629+0000 7f8fe661c080 20 get_system_obj_state: rctx=0x7fffc11fb930 obj=:.pools.avail state=0x55c147833220 s->prefetch_data=0 2021-03-02T12:03:28.635 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.629+0000 7f8fe661c080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-03-02T12:03:28.650 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.641+0000 7f8fe661c080 10 WARNING: store->list_zones() returned r=-2 2021-03-02T12:03:28.650 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.641+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.652 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.641+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.653 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.645+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.653 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.645+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.654 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.645+0000 7f8fe661c080 10 could not read realm id: (2) No such file or directory 2021-03-02T12:03:28.654 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.645+0000 7f8fe661c080 10 WARNING: failed to set zone as default, r=-22 2021-03-02T12:03:28.655 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.645+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.655 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.645+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.660 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.649+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.667 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados_obj.operate() r=0 bl.length=839 2021-03-02T12:03:28.668 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.668 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:28.668 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 zonegroup default 2021-03-02T12:03:28.669 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.669 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:28.670 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 10 Cannot find current period zone using local zone 2021-03-02T12:03:28.670 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 10 Using default name default 2021-03-02T12:03:28.671 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.671 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:28.672 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:28.672 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados_obj.operate() r=0 bl.length=839 2021-03-02T12:03:28.673 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 zone default found 2021-03-02T12:03:28.673 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:28.653+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:30.623 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:30.617+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:30.623 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:30.617+0000 7f8fe661c080 20 rados->read ofs=0 len=0 2021-03-02T12:03:30.625 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:30.621+0000 7f8fe661c080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:30.625 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:30.621+0000 7f8fe661c080 20 started sync module instance, tier type = 2021-03-02T12:03:30.625 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:30.621+0000 7f8fe661c080 20 started zone id=1656fb0b-b64e-4f5a-b23d-5171aabc5c41 (name=default) with tier type = 2021-03-02T12:03:32.684 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=0 2021-03-02T12:03:32.685 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=1 2021-03-02T12:03:32.685 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=2 2021-03-02T12:03:32.685 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=3 2021-03-02T12:03:32.685 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=4 2021-03-02T12:03:32.686 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=5 2021-03-02T12:03:32.686 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=6 2021-03-02T12:03:32.687 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 add_watcher() i=7 2021-03-02T12:03:32.687 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 2 all 8 watchers are set, enabling cache 2021-03-02T12:03:32.687 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:32.688 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fe661c080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:32.688 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:32.681+0000 7f8fa37de700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:36.642 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.637+0000 7f8fe661c080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-03-02T12:03:36.753 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.749+0000 7f8f11ffb700 20 reqs_thread_entry: start 2021-03-02T12:03:36.754 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.749+0000 7f8fe661c080 20 init_complete bucket index max shards: 11 2021-03-02T12:03:36.754 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.749+0000 7f8f077fe700 20 reqs_thread_entry: start 2021-03-02T12:03:37.194 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.749+0000 7f8f05ffb700 20 reqs_thread_entry: start 2021-03-02T12:03:37.198 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.773+0000 7f8fa37de700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:37.198 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.777+0000 7f8fe661c080 20 remove_watcher() i=0 2021-03-02T12:03:37.199 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.777+0000 7f8fe661c080 2 removed watcher, disabling cache 2021-03-02T12:03:37.199 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.777+0000 7f8fe661c080 20 remove_watcher() i=1 2021-03-02T12:03:37.199 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.781+0000 7f8fe661c080 20 remove_watcher() i=2 2021-03-02T12:03:37.200 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.785+0000 7f8fe661c080 20 remove_watcher() i=3 2021-03-02T12:03:37.200 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.785+0000 7f8fe661c080 20 remove_watcher() i=4 2021-03-02T12:03:37.200 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.789+0000 7f8fe661c080 20 remove_watcher() i=5 2021-03-02T12:03:37.201 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.789+0000 7f8fe661c080 20 remove_watcher() i=6 2021-03-02T12:03:37.201 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:36.793+0000 7f8fe661c080 20 remove_watcher() i=7 2021-03-02T12:03:37.201 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:03:37.202 DEBUG:tasks.util.rgw: json result: [] 2021-03-02T12:03:37.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:37.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:37.204 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-02T12:03:37.284 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:37.284 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:37.296 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.293+0000 7fbe91355080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:37.297 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.293+0000 7fbe91355080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:37.304 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbdc5ffb700 20 reqs_thread_entry: start 2021-03-02T12:03:37.304 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbe91355080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.306 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbe91355080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.306 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbe91355080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.307 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbe91355080 20 rados_obj.operate() r=0 bl.length=839 2021-03-02T12:03:37.308 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbe91355080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.308 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.301+0000 7fbe91355080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.308 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.305+0000 7fbe91355080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.310 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.305+0000 7fbe91355080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.310 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.305+0000 7fbe91355080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.310 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.305+0000 7fbe91355080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:37.319 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"1656fb0b-b64e-4f5a-b23d-5171aabc5c41","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:03:37.319 DEBUG:tasks.util.rgw: json result: {'id': '1656fb0b-b64e-4f5a-b23d-5171aabc5c41', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-02T12:03:37.320 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-03-02T12:03:37.320 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-03-02T12:03:37.320 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-03-02T12:03:37.397 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:37.397 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:37.409 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.405+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.877 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.405+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.878 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.405+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.878 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.409+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.878 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.409+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.879 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.409+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.879 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.409+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.879 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.409+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.879 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.425+0000 7fd324c67080 20 RGWRados::pool_iterate: got zone_info.1656fb0b-b64e-4f5a-b23d-5171aabc5c41 2021-03-02T12:03:37.879 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.425+0000 7fd324c67080 20 RGWRados::pool_iterate: got zonegroup_info.d841e95f-c257-4d5e-a6e4-77da853e936f 2021-03-02T12:03:37.880 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.425+0000 7fd324c67080 20 RGWRados::pool_iterate: got zone_names.default 2021-03-02T12:03:37.880 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.425+0000 7fd324c67080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-03-02T12:03:37.880 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.425+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.880 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.429+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.881 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.429+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.881 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.429+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=855 2021-03-02T12:03:37.881 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.429+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.881 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.881 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.882 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:37.883 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.883 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.884 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 10 cannot find current period zonegroup using local zonegroup 2021-03-02T12:03:37.884 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.884 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.884 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.885 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.885 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.885 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:37.885 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 zonegroup default 2021-03-02T12:03:37.885 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.886 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.886 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 10 Cannot find current period zone using local zone 2021-03-02T12:03:37.886 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.433+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.886 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.887 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.887 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:37.888 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.888 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados_obj.operate() r=0 bl.length=855 2021-03-02T12:03:37.888 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 zone default found 2021-03-02T12:03:37.888 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.888 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.889 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados->read ofs=0 len=0 2021-03-02T12:03:37.889 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:37.889 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 started sync module instance, tier type = 2021-03-02T12:03:37.889 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.437+0000 7fd324c67080 20 started zone id=1656fb0b-b64e-4f5a-b23d-5171aabc5c41 (name=default) with tier type = 2021-03-02T12:03:37.890 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=0 2021-03-02T12:03:37.890 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=1 2021-03-02T12:03:37.890 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=2 2021-03-02T12:03:37.890 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=3 2021-03-02T12:03:37.890 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=4 2021-03-02T12:03:37.891 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=5 2021-03-02T12:03:37.891 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=6 2021-03-02T12:03:37.891 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 add_watcher() i=7 2021-03-02T12:03:37.891 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 2 all 8 watchers are set, enabling cache 2021-03-02T12:03:37.892 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:37.892 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd324c67080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:37.892 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.469+0000 7fd2e1fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:37.892 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.553+0000 7fd253fff700 20 reqs_thread_entry: start 2021-03-02T12:03:37.893 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.553+0000 7fd251ffb700 20 reqs_thread_entry: start 2021-03-02T12:03:37.893 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.553+0000 7fd324c67080 20 init_complete bucket index max shards: 11 2021-03-02T12:03:37.893 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.553+0000 7fd23ffff700 20 reqs_thread_entry: start 2021-03-02T12:03:37.895 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.573+0000 7fd2e1fdb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-02T12:03:37.896 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.577+0000 7fd324c67080 20 remove_watcher() i=0 2021-03-02T12:03:37.896 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.577+0000 7fd324c67080 2 removed watcher, disabling cache 2021-03-02T12:03:37.896 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.577+0000 7fd324c67080 20 remove_watcher() i=1 2021-03-02T12:03:37.896 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.581+0000 7fd324c67080 20 remove_watcher() i=2 2021-03-02T12:03:37.897 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.585+0000 7fd324c67080 20 remove_watcher() i=3 2021-03-02T12:03:37.897 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.585+0000 7fd324c67080 20 remove_watcher() i=4 2021-03-02T12:03:37.897 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.589+0000 7fd324c67080 20 remove_watcher() i=5 2021-03-02T12:03:37.898 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.589+0000 7fd324c67080 20 remove_watcher() i=6 2021-03-02T12:03:37.898 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.593+0000 7fd324c67080 20 remove_watcher() i=7 2021-03-02T12:03:37.898 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:03:37.899 DEBUG:tasks.util.rgw: json result: [] 2021-03-02T12:03:37.899 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:37.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-02T12:03:37.900 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-02T12:03:37.980 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:37.980 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:37.993 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.989+0000 7f67de802080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-02T12:03:37.993 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.989+0000 7f67de802080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-02T12:03:38.001 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.997+0000 7f6712ffd700 20 reqs_thread_entry: start 2021-03-02T12:03:38.001 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.997+0000 7f67de802080 20 rados->read ofs=0 len=0 2021-03-02T12:03:38.002 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.997+0000 7f67de802080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:38.002 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:37.997+0000 7f67de802080 20 rados->read ofs=0 len=0 2021-03-02T12:03:38.003 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados_obj.operate() r=0 bl.length=855 2021-03-02T12:03:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados->read ofs=0 len=0 2021-03-02T12:03:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-02T12:03:38.004 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados->read ofs=0 len=0 2021-03-02T12:03:38.005 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados_obj.operate() r=0 bl.length=46 2021-03-02T12:03:38.005 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados->read ofs=0 len=0 2021-03-02T12:03:38.005 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:38.001+0000 7f67de802080 20 rados_obj.operate() r=0 bl.length=358 2021-03-02T12:03:38.016 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"1656fb0b-b64e-4f5a-b23d-5171aabc5c41","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:03:38.016 DEBUG:tasks.util.rgw: json result: {'id': '1656fb0b-b64e-4f5a-b23d-5171aabc5c41', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-02T12:03:38.017 INFO:tasks.rgw:Starting rgw... 2021-03-02T12:03:38.017 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-03-02T12:03:38.017 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-02T12:03:38.018 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-03-02T12:03:38.018 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-03-02T12:03:38.022 INFO:tasks.rgw.c1.client.0:Started 2021-03-02T12:03:38.023 INFO:tasks.rgw:rgw c1.client.1 config is {'port': 8001} 2021-03-02T12:03:38.023 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-02T12:03:38.023 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-03-02T12:03:38.024 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-03-02T12:03:38.031 INFO:tasks.rgw.c1.client.1:Started 2021-03-02T12:03:38.032 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-03-02T12:03:38.032 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-02T12:03:38.032 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-03-02T12:03:38.033 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-03-02T12:03:38.061 INFO:tasks.rgw.c2.client.0:Started 2021-03-02T12:03:38.062 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-03-02T12:03:38.062 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-02T12:03:38.062 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-02T12:03:38.063 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-02T12:03:38.065 INFO:tasks.rgw.c2.client.1:Started 2021-03-02T12:03:38.066 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi012.front.sepia.ceph.com:8000/ 2021-03-02T12:03:38.066 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000/ 2021-03-02T12:03:38.096 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:38.097 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:38.099 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8000: Connection refused 2021-03-02T12:03:38.100 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-02T12:03:39.102 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000/ 2021-03-02T12:03:39.122 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:39.122 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:39.127 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:03:39.129 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2021-03-02T12:03:39.130 INFO:tasks.rgw:Polling c1.client.1 until it starts accepting connections on http://smithi012.front.sepia.ceph.com:8001/ 2021-03-02T12:03:39.130 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001/ 2021-03-02T12:03:39.198 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:39.199 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:39.200 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:03:39.201 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2021-03-02T12:03:39.201 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi036.front.sepia.ceph.com:8000/ 2021-03-02T12:03:39.202 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000/ 2021-03-02T12:03:39.417 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:39.417 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:39.423 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 19454 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:03:39.424 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2021-03-02T12:03:39.424 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi036.front.sepia.ceph.com:8001/ 2021-03-02T12:03:39.425 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8001/ 2021-03-02T12:03:39.442 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:39.442 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:39.450 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:03:39.452 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2021-03-02T12:03:39.461 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-03-02T12:03:40.115 INFO:tasks.rgw_multisite:creating realm.. 2021-03-02T12:03:40.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:40.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:40.118 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:40.155 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:40.155 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:40.203 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","name":"test-realm","current_period":"fbfc7cb8-5f93-4552-919e-16e09cefcfbb","epoch":1} 2021-03-02T12:03:40.204 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-03-02T12:03:40.205 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:40.205 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:40.206 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:40.282 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:40.283 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:40.304 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:03:40.302+0000 7f243d90e080 -1 old format 2021-03-02T12:03:40.320 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}} 2021-03-02T12:03:40.320 INFO:tasks.rgw_multisite:creating master zone.. 2021-03-02T12:03:40.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:40.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:40.322 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --endpoints http://smithi012.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:40.412 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:40.412 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:40.483 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:03:40.483 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-03-02T12:03:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-03-02T12:03:41.557 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-03-02T12:03:42.552 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-03-02T12:03:42.577 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:42.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:42.582 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:42.616 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:42.617 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:42.644 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:03:42.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:42.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:42.647 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:42.730 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:42.731 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:50.778 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","epoch":1,"predecessor_uuid":"fbfc7cb8-5f93-4552-919e-16e09cefcfbb","sync_status":[],"period_map":{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":2} 2021-03-02T12:03:50.779 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-03-02T12:03:50.788 INFO:teuthology.orchestra.run:waiting for 300 2021-03-02T12:03:50.821 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2021-03-02T12:03:50.818+0000 7f5a19e85700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 13130) UID: 0 2021-03-02T12:03:50.822 INFO:tasks.rgw.c1.client.0.smithi012.stdout:2021-03-02T12:03:50.818+0000 7f5a315480c0 -1 shutting down 2021-03-02T12:03:54.312 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-03-02T12:03:56.890 INFO:tasks.rgw.c1.client.0:Stopped 2021-03-02T12:03:56.898 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-03-02T12:03:56.898 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-03-02T12:03:56.901 INFO:tasks.rgw.c1.client.0:Started 2021-03-02T12:03:56.901 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000 2021-03-02T12:03:56.918 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:56.918 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:56.921 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8000: Connection refused 2021-03-02T12:03:56.923 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-02T12:03:57.925 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8000 2021-03-02T12:03:57.943 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:03:57.943 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:03:57.948 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:03:57.949 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2021-03-02T12:03:57.949 INFO:tasks.rgw_multisite:creating admin user.. 2021-03-02T12:03:57.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:57.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:57.950 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:58.040 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:58.218 INFO:teuthology.orchestra.run.smithi012.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-03-02T12:03:58.246 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-03-02T12:03:58.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:58.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:58.248 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi012.front.sepia.ceph.com:8000 --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:58.292 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:58.292 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:58.408 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","name":"test-realm","current_period":"34dc26a2-30eb-4d82-a327-548c712f3e72","epoch":2} 2021-03-02T12:03:58.612 INFO:tasks.rgw_multisite:creating zone.. 2021-03-02T12:03:58.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:58.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:03:58.616 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --endpoints http://smithi036.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:03:58.663 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:03:58.663 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:03:58.718 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:58.713+0000 7f8cd6457080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:03:58.718 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:03:58.713+0000 7f8cd6457080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:03:58.730 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:03:58.810 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-03-02T12:03:59.643 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-03-02T12:03:59.657 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-03-02T12:04:00.021 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-03-02T12:04:00.674 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-03-02T12:04:00.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:00.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:00.686 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:00.729 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:00.732 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:00.759 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:04:00.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:00.760 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:00.760 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:00.843 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:00.843 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:08.801 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:04:09.888 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","epoch":2,"predecessor_uuid":"fbfc7cb8-5f93-4552-919e-16e09cefcfbb","sync_status":[],"period_map":{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":2} 2021-03-02T12:04:09.889 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-03-02T12:04:09.890 INFO:teuthology.orchestra.run:waiting for 300 2021-03-02T12:04:09.931 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:04:09.929+0000 7fefc2384700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 13306) UID: 0 2021-03-02T12:04:09.932 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:04:09.929+0000 7fefd9a470c0 -1 shutting down 2021-03-02T12:04:10.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-03-02T12:04:15.991 INFO:tasks.rgw.c2.client.0:Stopped 2021-03-02T12:04:15.999 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-03-02T12:04:16.000 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-03-02T12:04:16.003 INFO:tasks.rgw.c2.client.0:Started 2021-03-02T12:04:16.004 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000 2021-03-02T12:04:16.024 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:04:16.024 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:04:16.028 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi036.front.sepia.ceph.com port 8000: Connection refused 2021-03-02T12:04:16.029 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-02T12:04:16.339 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-03-02T12:04:17.031 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8000 2021-03-02T12:04:17.100 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:04:17.101 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:04:17.102 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:04:17.103 INFO:teuthology.orchestra.run.smithi036.stdout:anonymous 2021-03-02T12:04:17.104 INFO:tasks.rgw_multisite:creating zone.. 2021-03-02T12:04:17.104 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:17.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi012.front.sepia.ceph.com:8001', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:17.105 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --endpoints http://smithi012.front.sepia.ceph.com:8001 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:17.152 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:17.153 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:04:17.202+0000 7f9aa4761080 0 failed reading obj info from .rgw.root:zone_info.78024174-fc84-48b4-be73-8c69857e8755: (2) No such file or directory 2021-03-02T12:04:17.205 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:04:17.202+0000 7f9aa4761080 0 WARNING: could not read zone params for zone id=78024174-fc84-48b4-be73-8c69857e8755 name=test-zone2 2021-03-02T12:04:17.217 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:04:17.456 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c1 2021-03-02T12:04:18.635 INFO:teuthology.orchestra.run.smithi012.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-03-02T12:04:18.663 DEBUG:teuthology.orchestra.run.smithi012:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c1 2021-03-02T12:04:19.684 INFO:teuthology.orchestra.run.smithi012.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-03-02T12:04:19.702 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:19.709 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:19.710 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:19.746 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:19.774 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-02T12:04:19.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:19.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:19.776 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:19.854 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:19.855 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:27.854 INFO:teuthology.orchestra.run.smithi012.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:04:28.938 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","epoch":3,"predecessor_uuid":"fbfc7cb8-5f93-4552-919e-16e09cefcfbb","sync_status":[],"period_map":{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":2} 2021-03-02T12:04:28.940 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-03-02T12:04:28.948 INFO:teuthology.orchestra.run:waiting for 300 2021-03-02T12:04:28.985 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:04:28.982+0000 7f968effd700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground (PID: 13135) UID: 0 2021-03-02T12:04:28.985 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:04:28.982+0000 7f96aa3f50c0 -1 shutting down 2021-03-02T12:04:34.202 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-03-02T12:04:35.050 INFO:tasks.rgw.c1.client.1:Stopped 2021-03-02T12:04:35.089 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-03-02T12:04:35.090 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-03-02T12:04:35.095 INFO:tasks.rgw.c1.client.1:Started 2021-03-02T12:04:35.096 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001 2021-03-02T12:04:35.115 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:04:35.116 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:04:35.118 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8001: Connection refused 2021-03-02T12:04:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-02T12:04:36.122 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001 2021-03-02T12:04:36.145 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:04:36.146 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:04:36.151 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:04:36.152 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2021-03-02T12:04:36.153 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-03-02T12:04:36.153 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:36.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--url', 'http://smithi012.front.sepia.ceph.com:8000', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:36.155 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi012.front.sepia.ceph.com:8000 --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --url http://smithi012.front.sepia.ceph.com:8000 --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:36.206 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:36.207 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:36.250 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:04:36.245+0000 7fb71affd700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.18323e0d-bcde-4a6a-931a-d01d6665ad69.control err=-107 2021-03-02T12:04:36.278 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","name":"test-realm","current_period":"34dc26a2-30eb-4d82-a327-548c712f3e72","epoch":2} 2021-03-02T12:04:36.278 INFO:tasks.rgw_multisite:creating zone.. 2021-03-02T12:04:36.279 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:36.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--endpoints', 'http://smithi036.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:36.281 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --endpoints http://smithi036.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:36.331 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:36.332 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:36.392 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:04:36.389+0000 7f2275471080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:04:36.393 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:04:36.389+0000 7f2275471080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:04:36.393 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:04:36.389+0000 7f2275471080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:04:36.393 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:04:36.389+0000 7f2275471080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:04:36.403 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:04:36.404 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool create test-zone4.rgw.buckets.data 64 64 --cluster c2 2021-03-02T12:04:37.642 INFO:teuthology.orchestra.run.smithi036.stderr:pool 'test-zone4.rgw.buckets.data' created 2021-03-02T12:04:37.663 DEBUG:teuthology.orchestra.run.smithi036:> sudo ceph osd pool application enable test-zone4.rgw.buckets.data rgw --cluster c2 2021-03-02T12:04:38.686 INFO:teuthology.orchestra.run.smithi036.stderr:enabled application 'rgw' on pool 'test-zone4.rgw.buckets.data' 2021-03-02T12:04:38.760 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:39.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:39.229 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:39.272 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:39.273 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:39.305 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","domain_root":"test-zone4.rgw.meta:root","control_pool":"test-zone4.rgw.control","gc_pool":"test-zone4.rgw.log:gc","lc_pool":"test-zone4.rgw.log:lc","log_pool":"test-zone4.rgw.log","intent_log_pool":"test-zone4.rgw.log:intent","usage_log_pool":"test-zone4.rgw.log:usage","roles_pool":"test-zone4.rgw.meta:roles","reshard_pool":"test-zone4.rgw.log:reshard","user_keys_pool":"test-zone4.rgw.meta:users.keys","user_email_pool":"test-zone4.rgw.meta:users.email","user_swift_pool":"test-zone4.rgw.meta:users.swift","user_uid_pool":"test-zone4.rgw.meta:users.uid","otp_pool":"test-zone4.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone4.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone4.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone4.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2021-03-02T12:04:39.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:39.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:39.307 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:39.385 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:39.386 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:39.935 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-03-02T12:04:46.853 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:04:48.111 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","epoch":4,"predecessor_uuid":"fbfc7cb8-5f93-4552-919e-16e09cefcfbb","sync_status":[],"period_map":{"id":"34dc26a2-30eb-4d82-a327-548c712f3e72","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":2} 2021-03-02T12:04:48.112 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-03-02T12:04:48.120 INFO:teuthology.orchestra.run:waiting for 300 2021-03-02T12:04:48.155 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:04:48.153+0000 7fedb5ffb700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 13309) UID: 0 2021-03-02T12:04:48.156 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:04:48.153+0000 7fedd1c9d0c0 -1 shutting down 2021-03-02T12:04:50.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-03-02T12:04:54.225 INFO:tasks.rgw.c2.client.1:Stopped 2021-03-02T12:04:54.238 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-02T12:04:54.239 DEBUG:teuthology.orchestra.run.smithi036:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-02T12:04:54.249 INFO:tasks.rgw.c2.client.1:Started 2021-03-02T12:04:54.249 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8001 2021-03-02T12:04:54.276 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:04:54.277 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:04:54.277 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi036.front.sepia.ceph.com port 8001: Connection refused 2021-03-02T12:04:54.285 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-02T12:04:55.287 DEBUG:teuthology.orchestra.run.smithi036:> curl http://smithi036.front.sepia.ceph.com:8001 2021-03-02T12:04:55.304 INFO:teuthology.orchestra.run.smithi036.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:04:55.304 INFO:teuthology.orchestra.run.smithi036.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:04:55.309 INFO:teuthology.orchestra.run.smithi036.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 26800 2021-03-02T12:04:55.310 INFO:teuthology.orchestra.run.smithi036.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-00603e29e7-10e6-test-zone4","HostId":"10e6-test-zone4-test-zonegroup"} 2021-03-02T12:04:55.310 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-03-02T12:04:55.311 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-03-02T12:04:55.410 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-03-02T12:04:55.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:55.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:04:55.413 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:04:55.458 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:55.458 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:55.661 INFO:teuthology.orchestra.run.smithi012.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"080N5UR9YQ8R1YSJ48LO","secret_key":"sWPortH39iUbK10DwTa1zTOeg4rNAodUoL94wIUF"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-03-02T12:04:55.688 INFO:rgw_multi.tests:meta checkpoint 2021-03-02T12:04:55.688 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:04:55.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:04:55.689 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:04:55.771 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:55.772 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:55.930 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:04:55.960 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686695.652801_41.1","last_update":"2021-03-02T12:04:55.652801Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:04:55.961 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:04:55.962 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:04:55.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:04:55.963 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:04:56.004 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:04:56.004 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:04:56.210 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:04:56.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:04:56.651 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-03-02T12:05:01.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:01.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:01.223 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:01.264 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:01.429 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:01.465 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:01.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:06.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:06.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:06.469 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:06.510 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:06.510 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:06.696 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:06.696 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:11.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:11.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:11.708 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:11.748 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:11.749 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:11.946 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:11.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:16.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:16.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:16.971 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:17.013 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:17.013 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:17.210 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:17.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:22.214 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:22.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:22.227 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:22.269 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:22.270 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:22.482 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:22.516 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:27.517 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:27.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:27.527 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:27.568 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:27.568 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:27.784 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:27.784 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:32.786 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:32.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:32.796 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:32.837 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:32.837 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:33.040 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:33.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:38.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:38.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:38.127 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:38.168 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:38.169 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:38.378 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:38.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:43.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:43.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:43.381 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:43.425 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:43.425 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:43.638 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:43.809 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:48.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:48.817 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:48.818 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:48.859 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:48.859 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:49.055 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:49.056 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:54.057 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:54.062 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:54.063 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:54.103 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:54.104 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:54.297 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:54.298 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:05:59.301 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:59.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:05:59.315 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:05:59.358 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:05:59.359 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:05:59.565 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:05:59.566 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:04.567 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:04.581 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:04.582 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:04.625 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:04.626 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:04.826 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:04.827 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:09.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:09.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:09.841 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:09.884 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:09.884 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:10.094 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:10.095 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:15.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:15.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:15.108 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:15.150 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:15.150 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:15.342 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:15.401 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:20.402 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:20.410 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:20.411 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:20.457 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:20.457 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:20.665 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686695.652801_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:04:55.652801Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:20.666 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:06:20.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:06:20.667 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:20.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:20.668 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:20.712 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:20.712 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:20.917 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:20.918 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:25.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:25.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:25.961 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:26.003 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:26.205 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:26.205 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:31.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:31.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:31.208 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:31.252 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:31.253 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:31.458 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:31.458 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:36.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:36.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:36.470 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:36.515 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:36.516 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:36.723 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:36.797 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:41.799 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:41.806 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:41.806 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:41.848 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:41.849 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:42.041 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:42.042 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:47.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:47.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:47.053 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:47.093 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:47.094 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:47.295 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:47.553 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=0 master=1_1614686695.652801_41.1 target= 2021-03-02T12:06:52.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:52.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:52.557 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:52.602 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:52.603 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:52.802 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686695.652801_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:04:55.652801Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:52.803 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:06:52.803 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:06:52.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:52.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:52.804 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:52.851 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:52.851 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:53.112 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686695.652801_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:04:55.652801Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:53.113 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:06:53.113 INFO:rgw_multi.tests:meta checkpoint 2021-03-02T12:06:53.114 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.114 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.114 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:53.155 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:53.155 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:53.329 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:06:53.364 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686695.652801_41.1","last_update":"2021-03-02T12:04:55.652801Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:06:53.365 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:06:53.365 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.366 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:53.406 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:53.407 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:53.599 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686695.652801_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:04:55.652801Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:53.600 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:06:53.600 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:06:53.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.602 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:53.644 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:53.645 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:53.855 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686695.652801_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:04:55.652801Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:53.855 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:06:53.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:06:53.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:06:53.857 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:06:53.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:06:53.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:06:54.160 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686695.652801_41.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:04:55.652801Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:06:54.160 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:06:54.161 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on 'tasks.rgw_multi.tests' with args=[] 2021-03-02T12:06:54.171 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-1 2021-03-02T12:06:57.650 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-2 2021-03-02T12:07:01.735 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-3 2021-03-02T12:07:01.819 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:01.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:01.822 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:01.872 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:02.058 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:02.090 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686821.800003_54.1","last_update":"2021-03-02T12:07:01.800003Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:02.092 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:02.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:02.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:02.093 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:02.136 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:02.137 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:02.341 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686817.677429_50.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:06:57.677429Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:02.342 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686821.800003_54.1 target=1_1614686817.677429_50.1 2021-03-02T12:07:07.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:07.354 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:07.355 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:07.395 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:07.395 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:07.592 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686821.800003_54.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:01.800003Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:07.593 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:07.593 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:07.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:07.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:07.595 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:07.634 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:07.635 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:07.823 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686821.800003_54.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:01.800003Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:08.023 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:08.024 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:08.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:08.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:08.027 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:08.069 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:08.069 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:08.330 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686821.800003_54.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:01.800003Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:08.544 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:08.608 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_create ... ok 2021-03-02T12:07:08.612 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-4 2021-03-02T12:07:08.650 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-5 2021-03-02T12:07:08.704 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-6 2021-03-02T12:07:08.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:09.154 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:09.155 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:09.197 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:09.367 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:09.495 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686828.731624_68.1","last_update":"2021-03-02T12:07:08.731624Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:09.496 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:09.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:09.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:09.498 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:09.543 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:09.543 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:09.772 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686828.625039_56.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:08.625039Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:09.938 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686828.731624_68.1 target=1_1614686828.625039_56.1 2021-03-02T12:07:14.939 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:14.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:14.949 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:14.991 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:14.991 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:15.270 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686828.731624_68.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:08.731624Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:15.270 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:15.271 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:15.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:15.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:15.274 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:15.317 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:15.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:15.524 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686828.731624_68.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:08.731624Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:15.525 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:15.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:15.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:15.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:15.527 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:15.568 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:15.568 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:15.832 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686828.731624_68.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:08.731624Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:15.833 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:16.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:16.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:16.600 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:16.636 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:16.804 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:16.836 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686836.465803_108.1","last_update":"2021-03-02T12:07:16.465803Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:16.837 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:16.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:16.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:16.839 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:16.880 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:16.881 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:17.079 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686828.731624_68.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:08.731624Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:17.080 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686836.465803_108.1 target=1_1614686828.731624_68.1 2021-03-02T12:07:22.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:22.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:22.094 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:22.134 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:22.135 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:22.353 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686836.465803_108.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:16.465803Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:22.358 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:22.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:22.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:22.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:22.366 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:22.429 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:22.430 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:22.651 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686836.465803_108.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:16.465803Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:22.652 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:22.652 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:22.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:22.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:22.654 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:22.699 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:22.700 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:22.960 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686836.465803_108.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:16.465803Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:22.961 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:23.029 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_recreate ... ok 2021-03-02T12:07:23.031 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-7 2021-03-02T12:07:23.069 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-8 2021-03-02T12:07:23.132 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-9 2021-03-02T12:07:23.192 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:23.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:23.194 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:23.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:23.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:23.410 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:23.444 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686843.170580_122.1","last_update":"2021-03-02T12:07:23.170580Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:23.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:23.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:23.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:23.447 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:23.500 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:23.500 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:23.695 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686836.465803_108.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:16.465803Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:23.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686843.170580_122.1 target=1_1614686836.465803_108.1 2021-03-02T12:07:28.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:28.706 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:28.707 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:28.748 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:28.748 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:28.946 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686843.170580_122.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:23.170580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:28.947 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:28.948 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:28.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:28.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:28.949 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:28.991 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:28.992 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:29.217 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686843.170580_122.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:23.170580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:29.317 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:29.320 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:29.320 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:29.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:29.321 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:29.363 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:29.629 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:29.631 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686843.170580_122.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:23.170580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:29.631 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:29.862 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:29.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:29.873 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:29.912 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:30.098 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:30.346 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:30.346 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:30.347 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:30.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:30.348 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:30.390 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:30.390 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:30.585 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686843.170580_122.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:23.170580Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:30.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1614686849.835843_128.1 target=1_1614686843.170580_122.1 2021-03-02T12:07:35.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:35.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:35.835 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:35.875 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:35.876 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:36.074 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:36.074 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:36.075 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:36.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:36.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:36.077 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:36.117 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:36.117 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:36.354 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:36.354 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:36.355 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:36.355 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:36.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:36.356 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:36.400 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:36.400 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:36.767 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:36.768 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:36.785 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_remove ... ok 2021-03-02T12:07:36.788 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-10 2021-03-02T12:07:36.820 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-11 2021-03-02T12:07:36.872 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-12 2021-03-02T12:07:37.527 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:37.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:37.528 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:37.571 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:37.572 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:37.736 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:37.767 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614686856.903336_59.1","last_update":"2021-03-02T12:07:36.903336Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:37.768 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:37.768 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:37.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:37.770 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:37.811 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:37.812 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:38.028 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686856.903336_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:36.903336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:38.029 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:38.029 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:38.030 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.031 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:38.071 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:38.071 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:38.264 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686856.903336_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:36.903336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:38.264 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:38.265 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:38.265 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.267 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:38.305 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:38.306 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:38.556 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686856.903336_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:36.903336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:38.557 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:38.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.558 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:38.600 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:38.601 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:38.819 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.11.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:38.819 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-10 2021-03-02T12:07:38.820 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:38.821 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:38.866 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:39.107 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:39.109 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:39.289 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-10 2021-03-02T12:07:39.289 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-10 zones={test-zone2, test-zone1} 2021-03-02T12:07:39.301 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:39.313 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:39.314 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:39.314 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:39.314 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:39.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:39.322 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:39.323 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:39.323 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:39.324 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:39.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:39.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:39.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:39.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:39.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:39.837 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:39.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:40.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:40.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:40.269 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:40.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:40.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:40.596 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-10 zones={test-zone2, test-zone1} 2021-03-02T12:07:40.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:40.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:40.663 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:40.705 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:40.705 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:40.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.11.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:40.949 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-10 2021-03-02T12:07:40.950 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:40.951 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:40.951 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:41.003 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:41.003 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:41.228 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:41.229 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-10 2021-03-02T12:07:41.229 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-10 zones={test-zone3, test-zone1} 2021-03-02T12:07:41.239 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:41.246 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:41.247 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:41.247 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:41.248 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:41.248 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:41.254 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:41.255 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:41.268 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:41.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:41.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:41.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:41.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:41.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:41.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:42.016 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:42.132 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:42.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:42.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:42.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:42.598 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-10 zones={test-zone3, test-zone1} 2021-03-02T12:07:42.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:42.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:42.600 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:42.640 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:42.640 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:42.868 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.17.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.10.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.10.5"},{"key":8,"val":"00000000001.11.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:42.868 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-10 2021-03-02T12:07:42.869 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:42.869 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:42.870 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:42.915 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:42.915 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:43.178 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.17.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.10.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.10.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.11.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:43.179 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-10 2021-03-02T12:07:43.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:43.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:43.180 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:43.262 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:43.440 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:43.459 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.8.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.9.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:43.460 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-11 2021-03-02T12:07:43.460 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:43.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:43.461 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:43.509 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:43.509 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:43.732 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:43.733 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-11 2021-03-02T12:07:43.733 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-11 zones={test-zone1, test-zone2} 2021-03-02T12:07:43.752 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:43.769 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:43.770 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:43.770 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:43.771 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:43.771 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:43.783 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:43.783 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:43.784 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:43.784 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:43.807 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:43.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:44.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:44.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:44.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:44.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:44.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:44.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:44.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:44.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:44.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:44.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:45.075 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-11 zones={test-zone1, test-zone2} 2021-03-02T12:07:45.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:45.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:45.077 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:45.121 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:45.122 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:45.324 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.8.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.8.5"},{"key":8,"val":"00000000001.9.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:45.325 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-11 2021-03-02T12:07:45.325 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:45.326 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:45.326 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:45.373 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:45.602 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.9.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:45.602 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-11 2021-03-02T12:07:45.603 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-11 zones={test-zone3, test-zone2} 2021-03-02T12:07:45.614 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:45.621 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:45.622 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:45.622 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:45.623 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:45.623 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:45.630 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:45.631 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:45.631 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:45.631 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:45.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:45.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:45.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:45.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:45.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:46.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:46.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:46.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:46.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:46.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:46.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:46.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:46.752 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-11 zones={test-zone3, test-zone2} 2021-03-02T12:07:46.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:46.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:46.756 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:46.799 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:46.799 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:47.014 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.3.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.7.5"},{"key":8,"val":"00000000001.11.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:47.015 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-12 2021-03-02T12:07:47.020 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:47.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:47.021 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:47.102 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:47.332 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.3.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:47.333 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-12 2021-03-02T12:07:47.333 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-12 zones={test-zone1, test-zone3} 2021-03-02T12:07:47.345 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:47.351 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:47.352 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:47.352 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:47.353 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:47.353 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:47.360 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:47.360 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:47.361 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:47.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:47.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:47.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:47.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:47.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:47.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:47.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:48.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:48.134 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:48.247 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:48.303 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:48.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:48.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-12 zones={test-zone1, test-zone3} 2021-03-02T12:07:48.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:48.537 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:48.538 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:48.581 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:48.581 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:48.784 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.3.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.7.5"},{"key":8,"val":"00000000001.11.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:48.784 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-12 2021-03-02T12:07:48.785 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:48.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:48.786 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:48.833 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:48.834 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:49.050 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.3.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:49.050 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-12 2021-03-02T12:07:49.051 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-12 zones={test-zone2, test-zone3} 2021-03-02T12:07:49.061 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:49.068 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:49.068 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:49.069 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:49.069 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:49.070 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:49.079 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-02T12:07:49.080 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-02T12:07:49.081 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-02T12:07:49.082 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:49.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:49.152 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:49.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-02T12:07:49.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:49.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:49.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-02T12:07:49.669 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:49.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:49.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-02T12:07:49.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:50.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:50.133 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:50.239 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-12 zones={test-zone2, test-zone3} 2021-03-02T12:07:50.390 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_sync ... ok 2021-03-02T12:07:50.396 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-13 2021-03-02T12:07:50.430 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-14 2021-03-02T12:07:50.485 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-15 2021-03-02T12:07:50.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:50.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:50.752 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:50.799 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:50.968 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:07:51.027 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614686870.523377_77.1","last_update":"2021-03-02T12:07:50.523377Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:07:51.027 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:07:51.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:51.028 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:51.029 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:51.071 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:51.072 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:51.264 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686856.903336_59.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:36.903336Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:51.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614686870.523377_77.1 target=1_1614686856.903336_59.1 2021-03-02T12:07:56.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:56.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:56.418 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:56.459 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:56.460 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:56.672 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686870.523377_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:50.523377Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:56.672 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:07:56.673 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:07:56.673 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:56.674 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:56.674 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:56.717 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:56.718 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:56.922 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686870.523377_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:50.523377Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:56.923 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:07:56.923 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:07:56.924 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:56.924 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:56.925 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:56.966 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:56.966 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:57.230 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686870.523377_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:50.523377Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:07:57.230 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:07:57.231 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:57.232 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:57.232 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:57.277 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:57.277 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:57.476 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:57.476 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:07:57.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:57.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:57.478 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:57.522 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:57.522 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:57.750 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:57.751 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:07:57.752 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-13 zones={test-zone2, test-zone1} 2021-03-02T12:07:57.764 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:57.811 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:57.811 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:57.819 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:57.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:57.931 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:58.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:58.155 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-13 zones={test-zone2, test-zone1} 2021-03-02T12:07:58.156 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:58.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:58.157 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:58.200 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:58.201 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:58.404 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:58.405 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:07:58.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:58.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:58.408 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:58.490 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:58.701 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:58.702 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:07:58.702 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-13 zones={test-zone3, test-zone1} 2021-03-02T12:07:58.712 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:07:58.759 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:58.759 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:07:58.768 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:07:58.831 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:58.889 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:59.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:07:59.116 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-13 zones={test-zone3, test-zone1} 2021-03-02T12:07:59.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.117 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:59.156 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:59.157 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:59.366 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:59.366 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:07:59.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.368 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:59.413 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:59.413 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:59.676 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:07:59.677 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:07:59.678 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.679 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:59.724 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:59.725 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:07:59.919 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.12.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:07:59.919 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:07:59.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:07:59.921 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:07:59.973 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:07:59.974 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:00.170 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:00.171 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:00.171 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-14 zones={test-zone1, test-zone2} 2021-03-02T12:08:00.183 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:00.235 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:00.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:00.241 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:00.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:00.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:00.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:00.577 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-14 zones={test-zone1, test-zone2} 2021-03-02T12:08:00.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:00.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:00.579 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:00.621 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:00.621 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:00.814 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.12.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:00.814 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:00.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:00.815 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:00.816 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:00.860 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:00.861 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:01.092 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:01.787 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:01.787 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-14 zones={test-zone3, test-zone2} 2021-03-02T12:08:01.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:01.847 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:01.847 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:01.853 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:01.908 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:01.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:02.090 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:02.192 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-14 zones={test-zone3, test-zone2} 2021-03-02T12:08:02.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:02.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:02.194 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:02.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:02.355 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:02.446 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.10.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:02.447 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:02.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:02.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:02.450 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:02.537 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:02.556 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:02.763 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:02.764 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:02.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-15 zones={test-zone1, test-zone3} 2021-03-02T12:08:02.775 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:02.827 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:02.975 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:02.983 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:03.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:03.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:03.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:03.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-15 zones={test-zone1, test-zone3} 2021-03-02T12:08:03.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:03.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:03.584 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:03.626 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:03.627 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:03.827 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.10.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:03.954 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:03.955 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:03.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:03.956 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:04.002 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:04.003 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:04.229 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:04.443 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:04.444 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-15 zones={test-zone2, test-zone3} 2021-03-02T12:08:04.456 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:04.507 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:04.507 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:04.515 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-02T12:08:04.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:04.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:04.752 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-02T12:08:05.084 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-15 zones={test-zone2, test-zone3} 2021-03-02T12:08:05.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:05.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:05.124 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:05.197 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:05.198 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:05.403 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.18.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:05.404 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:08:05.404 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:05.405 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:05.405 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:05.454 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:05.455 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:05.670 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:05.672 WARNING:rgw_multi.tests:bucket majrnq-13 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.18.5 target=00000000001.16.5 2021-03-02T12:08:10.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:10.685 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:10.686 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:10.732 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:10.733 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:10.929 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:10.930 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:08:10.931 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-13 zones={test-zone2, test-zone1} 2021-03-02T12:08:10.941 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:10.995 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:11.064 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-13 zones={test-zone2, test-zone1} 2021-03-02T12:08:11.065 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.066 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:11.106 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:11.306 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.18.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:11.307 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:08:11.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.308 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:11.392 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:11.393 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:11.604 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:11.605 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:08:11.606 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-13 zones={test-zone3, test-zone1} 2021-03-02T12:08:11.615 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:11.667 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:11.740 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-13 zones={test-zone3, test-zone1} 2021-03-02T12:08:11.741 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.742 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.743 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:11.786 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:11.786 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:11.988 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.18.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:11.989 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:08:11.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:11.990 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:12.036 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:12.036 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:12.300 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.18.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:12.301 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-13 2021-03-02T12:08:12.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:12.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:12.318 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:12.384 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:12.384 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:12.581 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:12.581 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:12.582 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:12.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:12.584 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:12.631 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:12.631 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:12.848 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.12.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:12.906 WARNING:rgw_multi.tests:bucket majrnq-14 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.16.5 target=00000000001.12.5 2021-03-02T12:08:17.907 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:17.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:17.919 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:17.963 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:17.964 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:18.172 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:18.173 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:18.173 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-14 zones={test-zone1, test-zone2} 2021-03-02T12:08:18.183 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:18.231 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:18.293 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-14 zones={test-zone1, test-zone2} 2021-03-02T12:08:18.294 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:18.294 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:18.295 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:18.334 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:18.334 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:18.532 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.16.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:18.532 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:18.533 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:18.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:18.534 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:18.582 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:18.787 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:18.788 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-14 2021-03-02T12:08:18.789 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-14 zones={test-zone3, test-zone2} 2021-03-02T12:08:18.801 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:18.851 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:18.912 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-14 zones={test-zone3, test-zone2} 2021-03-02T12:08:18.930 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:18.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:18.931 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:18.971 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:18.972 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:19.167 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:19.167 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:19.168 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:19.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:19.169 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:19.252 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:19.469 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.10.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:19.469 WARNING:rgw_multi.tests:bucket majrnq-15 zone test-zone1 behind zone test-zone3: shard=1 master=00000000002.15.5 target=00000000001.10.5 2021-03-02T12:08:24.470 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:24.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:24.472 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:24.518 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:24.518 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:24.733 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:24.831 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:24.832 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-15 zones={test-zone1, test-zone3} 2021-03-02T12:08:24.844 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:24.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:25.603 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-15 zones={test-zone1, test-zone3} 2021-03-02T12:08:25.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:25.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:25.607 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:25.649 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:25.650 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:25.828 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.15.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:25.853 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:25.853 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:25.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:25.854 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:25.900 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:25.900 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:26.107 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.15.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:26.125 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-15 2021-03-02T12:08:26.126 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-15 zones={test-zone2, test-zone3} 2021-03-02T12:08:26.137 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:08:26.182 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:08:26.258 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-15 zones={test-zone2, test-zone3} 2021-03-02T12:08:26.259 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_object_delete ... ok 2021-03-02T12:08:26.263 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-16 2021-03-02T12:08:26.299 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-17 2021-03-02T12:08:26.358 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-18 2021-03-02T12:08:26.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:26.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:26.498 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:26.558 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:26.559 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:26.724 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:08:26.753 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614686906.489696_108.1","last_update":"2021-03-02T12:08:26.489696Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:08:26.754 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:08:26.754 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:26.755 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:26.755 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:26.806 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:26.806 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:27.000 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686870.523377_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:07:50.523377Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:08:27.000 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614686906.489696_108.1 target=1_1614686870.523377_77.1 2021-03-02T12:08:32.002 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.002 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.003 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:32.045 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:32.045 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:32.233 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686906.489696_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:08:26.489696Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:08:32.234 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:08:32.235 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:08:32.235 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.236 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:32.279 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:32.280 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:32.478 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686906.489696_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:08:26.489696Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:08:32.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:08:32.479 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:08:32.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.481 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:32.521 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:32.522 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:32.784 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686906.489696_108.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:08:26.489696Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:08:32.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:08:32.835 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:32.836 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:32.876 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:33.059 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.13.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:33.059 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:33.060 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:33.060 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:33.061 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:33.105 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:33.106 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:33.304 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:33.305 WARNING:rgw_multi.tests:bucket majrnq-16 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.13.10 target= 2021-03-02T12:08:38.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:38.309 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:38.313 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:38.364 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:38.364 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:38.570 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:38.571 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:38.572 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:38.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:38.573 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:38.621 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:38.622 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:38.815 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:38.815 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:38.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:38.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:38.817 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:38.901 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:38.902 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:39.115 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:39.116 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:39.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.118 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:39.199 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:39.200 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:39.397 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:39.397 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:39.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.399 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:39.444 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:39.445 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:39.709 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:39.710 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:39.759 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.759 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.760 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:39.801 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:39.802 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:39.989 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:39.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:08:39.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:39.991 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:40.037 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:40.244 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:40.245 WARNING:rgw_multi.tests:bucket majrnq-17 zone test-zone1 behind zone test-zone2: shard=5 master=00000000002.14.10 target= 2021-03-02T12:08:45.245 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:45.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:45.262 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:45.307 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:45.308 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:45.513 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:45.628 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:08:45.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:45.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:45.629 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:45.671 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:45.671 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:45.868 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:45.962 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:08:45.963 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:45.963 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:45.964 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:46.009 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:46.064 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:46.220 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000005.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:46.221 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:08:46.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:46.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:46.331 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:46.372 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:46.373 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:46.574 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.11.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:46.631 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:08:46.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:46.633 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:46.633 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:46.678 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:46.679 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:46.885 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:46.886 WARNING:rgw_multi.tests:bucket majrnq-18 zone test-zone1 behind zone test-zone3: shard=5 master=00000000002.11.10 target= 2021-03-02T12:08:51.887 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:51.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:51.889 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:51.932 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:51.933 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:52.168 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000005.17.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:52.168 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:08:52.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:52.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:52.170 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:52.252 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:52.252 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:52.472 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.17.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:52.472 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:08:52.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:52.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:52.474 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:52.519 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:52.520 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:52.715 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000005.17.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:52.716 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:08:54.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:54.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:54.490 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:54.530 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:54.751 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.32.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.16.10"},{"key":4,"val":""},{"key":5,"val":"00000000007.23.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:08:54.751 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:08:54.752 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:54.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:54.753 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:08:54.801 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:08:54.908 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:08:54.992 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000005.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:08:54.992 WARNING:rgw_multi.tests:bucket majrnq-16 zone test-zone2 behind zone test-zone1: shard=3 master=00000000002.16.10 target=, shard=5 master=00000000007.23.10 target=00000000005.19.9 2021-03-02T12:08:59.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:59.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:08:59.995 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:00.042 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:00.042 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:00.238 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000008.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:00.239 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:00.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:00.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:00.241 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:00.282 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:00.487 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.32.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:00.488 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:00.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:00.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:00.489 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:00.573 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:00.574 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:00.810 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000008.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:00.811 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:00.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:00.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:00.813 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:00.896 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:00.896 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:01.117 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.32.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:01.117 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:01.118 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.119 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:01.163 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:01.163 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:01.425 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.18.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:01.426 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:01.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.427 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.428 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:01.513 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:01.514 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:01.709 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.24.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:01.709 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:01.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.711 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:01.756 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:01.757 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:01.981 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000007.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:01.981 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:01.982 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.982 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:01.983 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:02.024 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:02.024 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:02.227 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.24.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.27.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:02.228 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:02.228 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:02.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:02.229 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:02.275 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:02.276 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:02.497 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000003.24.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000010.27.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000007.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:02.498 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:02.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:02.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:02.499 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:02.541 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:02.542 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:02.744 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.23.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.39.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:02.744 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:02.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:02.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:02.746 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:02.831 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:03.049 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000014.39.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:03.049 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:03.050 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:03.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:03.051 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:03.091 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:03.091 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:03.307 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.23.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.39.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:03.307 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:03.308 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:03.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:03.309 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:03.355 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:03.356 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:03.560 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000014.39.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:03.561 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:03.562 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-16 zones={test-zone2, test-zone1} 2021-03-02T12:09:03.574 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:03.583 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:03.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:03.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:03.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:03.584 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:03.593 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:03.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:03.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:03.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:03.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:03.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:04.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:04.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:04.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:04.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:04.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:04.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:04.884 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:04.992 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:05.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:05.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:05.483 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-16 zones={test-zone2, test-zone1} 2021-03-02T12:09:05.484 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-16 zones={test-zone3, test-zone1} 2021-03-02T12:09:05.493 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:05.499 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:05.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:05.500 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:05.501 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:05.501 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:05.555 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:05.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:05.556 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:05.557 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:05.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:05.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:05.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:05.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:06.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:06.305 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:06.421 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:06.476 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:06.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:06.724 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:06.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:06.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:07.022 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-16 zones={test-zone3, test-zone1} 2021-03-02T12:09:07.023 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-16 zones={test-zone3, test-zone2} 2021-03-02T12:09:07.033 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:07.041 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:07.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:07.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:07.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:07.043 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:07.050 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:07.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:07.051 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:07.052 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:07.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:07.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:07.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:07.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:07.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:07.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:07.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:07.666 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:07.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:07.898 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:07.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:08.070 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:08.181 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-16 zones={test-zone3, test-zone2} 2021-03-02T12:09:08.182 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.185 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:08.225 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:08.268 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:08.443 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.53.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:08.488 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:08.488 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.490 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:08.535 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:08.535 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:08.731 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000010.45.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000003.53.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:08.732 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:08.732 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.734 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:08.782 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:08.783 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:08.992 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.53.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:08.993 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:08.993 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:08.994 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:09.087 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:09.088 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:09.305 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000010.45.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000003.53.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:09.306 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:09.307 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:09.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:09.308 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:09.396 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:09.616 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.45.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.53.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:09.617 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:09.617 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:09.618 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:09.619 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:09.675 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:09.675 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:10.040 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.45.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.53.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:10.041 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:10.041 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.042 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:10.134 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:10.134 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:10.327 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.45.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:10.328 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:10.329 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.329 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.330 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:10.376 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:10.377 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:10.601 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000003.45.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000010.39.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000008.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:10.602 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:10.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.604 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:10.647 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:10.648 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:10.829 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.45.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.39.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:10.830 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:10.830 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:10.831 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:10.877 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:11.127 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000003.45.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000010.39.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000008.26.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:11.127 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:11.128 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.128 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.129 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:11.210 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:11.440 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.28.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.30.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:11.441 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:11.441 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.442 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.442 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:11.528 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:11.528 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:11.744 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000003.30.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000014.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:11.744 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:11.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.746 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:11.784 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:11.992 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.28.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.30.9"},{"key":4,"val":""},{"key":5,"val":"00000000014.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:11.992 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:11.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:11.994 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:12.041 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:12.041 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:12.241 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000003.30.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000014.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:12.241 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:12.242 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-17 zones={test-zone2, test-zone1} 2021-03-02T12:09:12.252 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:12.259 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:12.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:12.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:12.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:12.262 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:12.271 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:12.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:12.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:12.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:12.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:12.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:12.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:12.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:12.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:12.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:12.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:12.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:12.921 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:13.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:13.091 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:13.205 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:13.311 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-17 zones={test-zone2, test-zone1} 2021-03-02T12:09:13.312 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-17 zones={test-zone3, test-zone1} 2021-03-02T12:09:13.321 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:13.327 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:13.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:13.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:13.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:13.329 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:13.335 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:13.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:13.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:13.336 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:13.350 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:13.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:13.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:13.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:13.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:13.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:13.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:13.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:13.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:14.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:14.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:14.270 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:14.375 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-17 zones={test-zone3, test-zone1} 2021-03-02T12:09:14.375 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-17 zones={test-zone3, test-zone2} 2021-03-02T12:09:14.385 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:14.392 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:14.392 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:14.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:14.393 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:14.394 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:14.400 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:14.401 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:14.415 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:14.425 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:14.491 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:14.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:14.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:14.728 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:14.833 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:14.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:15.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:15.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:15.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:15.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:15.405 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-17 zones={test-zone3, test-zone2} 2021-03-02T12:09:15.406 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:15.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:15.408 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:15.451 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:15.451 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:15.655 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.52.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:15.656 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:15.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:15.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:15.659 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:15.705 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:15.705 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:15.897 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000010.52.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000007.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:15.958 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:15.966 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:15.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:15.967 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:16.009 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:16.033 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:16.236 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.52.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:16.361 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:16.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:16.362 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:16.363 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:16.408 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:16.408 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:16.642 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000010.52.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000007.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:16.643 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:16.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:16.644 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:16.645 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:16.688 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:16.688 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:16.905 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.52.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.15.9"},{"key":4,"val":""},{"key":5,"val":"00000000007.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:16.905 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:16.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:16.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:16.913 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:16.957 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:17.010 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:17.219 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.52.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.15.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:17.320 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:17.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:17.322 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:17.323 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:17.365 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:17.366 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:17.569 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.26.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.25.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:17.570 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:17.571 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:17.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:17.573 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:17.618 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:17.619 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:17.847 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000003.26.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:17.899 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:17.900 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:17.900 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:17.901 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:17.945 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:17.946 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:18.146 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.26.9"},{"key":2,"val":""},{"key":3,"val":"00000000010.25.5"},{"key":4,"val":""},{"key":5,"val":"00000000007.21.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:18.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:18.167 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:18.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:18.170 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:18.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:18.313 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:18.445 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000003.26.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000010.25.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000007.21.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:18.497 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:18.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:18.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:18.499 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:18.543 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:18.544 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:18.736 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.22.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000015.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:18.762 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:18.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:18.763 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:18.764 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:18.822 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:18.823 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:19.048 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000015.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:19.049 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:19.049 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:19.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:19.050 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:19.134 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:19.298 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:19.355 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.22.9"},{"key":2,"val":""},{"key":3,"val":"00000000003.14.9"},{"key":4,"val":""},{"key":5,"val":"00000000015.37.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:19.356 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:19.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:19.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:19.357 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:19.419 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:19.419 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:19.633 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000003.14.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000015.37.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:19.635 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:19.635 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-18 zones={test-zone2, test-zone1} 2021-03-02T12:09:19.650 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:19.664 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:19.665 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:19.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:19.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:19.667 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:19.696 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:19.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:19.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:19.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:19.726 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:19.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:19.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:19.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:19.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:20.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:20.221 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:20.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:20.430 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:20.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:20.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:20.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:20.913 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-18 zones={test-zone2, test-zone1} 2021-03-02T12:09:20.914 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-18 zones={test-zone3, test-zone1} 2021-03-02T12:09:20.930 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:20.944 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:20.944 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:20.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:20.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:20.946 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:20.959 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:20.960 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:20.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:20.961 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:20.986 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:21.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:21.088 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:21.161 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:21.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:21.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:21.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:21.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:21.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:21.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:21.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:21.933 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:22.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-18 zones={test-zone3, test-zone1} 2021-03-02T12:09:22.048 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-18 zones={test-zone3, test-zone2} 2021-03-02T12:09:22.059 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:22.068 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:22.069 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:22.073 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:22.074 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:22.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:22.085 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:22.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:22.086 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:22.087 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:22.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:22.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:22.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:22.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:22.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:22.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:22.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:22.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:22.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:22.809 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:22.867 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:22.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:23.097 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-18 zones={test-zone3, test-zone2} 2021-03-02T12:09:23.725 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:23.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:23.726 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:23.766 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:23.767 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:23.967 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.62.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.18.9"},{"key":4,"val":""},{"key":5,"val":"00000000008.25.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:23.967 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:23.968 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:23.968 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:23.969 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:24.015 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:24.016 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:24.218 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000010.32.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.9"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000008.25.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:24.219 WARNING:rgw_multi.tests:bucket majrnq-16 zone test-zone2 behind zone test-zone1: shard=1 master=00000000011.62.5 target=00000000010.32.5 2021-03-02T12:09:29.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:29.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:29.225 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:29.271 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:29.271 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:29.496 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000011.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:29.497 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:29.498 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:29.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:29.499 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:29.541 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:29.740 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.62.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.21.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:29.770 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:29.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:29.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:29.772 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:29.826 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:30.047 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000011.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000004.21.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000009.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:30.048 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:30.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.049 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:30.134 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:30.135 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:30.349 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.62.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.21.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.28.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:30.349 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:30.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.351 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:30.422 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:30.424 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:30.682 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.21.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:30.683 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-16 2021-03-02T12:09:30.683 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.685 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:30.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:30.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:30.968 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.28.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:30.969 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:30.969 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.970 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:30.970 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:31.018 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:31.019 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:31.247 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000004.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000011.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000008.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:31.248 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:31.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:31.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:31.250 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:31.300 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:31.301 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:31.513 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.28.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:31.513 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:31.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:31.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:31.515 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:31.566 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:31.566 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:31.776 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000004.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000011.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000008.28.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:31.777 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-16 2021-03-02T12:09:31.789 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:31.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:31.790 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:31.868 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:31.868 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:32.075 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.42.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:32.075 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:32.076 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:32.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:32.077 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:32.167 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:32.396 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000015.42.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:32.397 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:32.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:32.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:32.399 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:32.478 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:32.478 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:32.690 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.19.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.42.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:32.691 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:32.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:32.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:32.693 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:32.740 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:32.741 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:32.976 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"vbbyfnWB05c.ZOdz4yBL5CRvQxYv547","ns":""},"count":1},"inc_marker":{"position":"00000000004.27.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"gXWWepGuOrekIkYYU6B12jcb1j.CxKf","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"u.WD5kZW48.Ks-skwypOoWa2tirw1K6","ns":""},"count":1},"inc_marker":{"position":"00000000015.42.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:32.977 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-16 2021-03-02T12:09:32.979 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-16 zones={test-zone2, test-zone1} 2021-03-02T12:09:32.992 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:32.999 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:33.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:33.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:33.000 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:33.001 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:33.011 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:33.012 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:33.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:33.015 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:33.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:33.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:33.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:33.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:33.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:33.357 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:33.474 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:33.532 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:33.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:33.766 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:33.827 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:33.948 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:34.056 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-16 zones={test-zone2, test-zone1} 2021-03-02T12:09:34.057 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-16 zones={test-zone3, test-zone1} 2021-03-02T12:09:34.068 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:34.074 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:34.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:34.075 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:34.076 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:34.076 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:34.082 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:34.082 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:34.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:34.083 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:34.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:34.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:34.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:34.240 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:34.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:34.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:34.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:34.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:34.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:34.805 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:34.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:34.979 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:35.089 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-16 zones={test-zone3, test-zone1} 2021-03-02T12:09:35.090 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-16 zones={test-zone3, test-zone2} 2021-03-02T12:09:35.102 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:35.113 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:35.114 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:35.115 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:35.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:35.116 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:35.125 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:35.125 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:35.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:35.126 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:35.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:35.147 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:35.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:35.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:35.337 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:35.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:35.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:35.613 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:35.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:35.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:35.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:36.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:36.121 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-16 zones={test-zone3, test-zone2} 2021-03-02T12:09:36.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.123 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:36.169 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:36.169 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:36.366 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.57.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.59.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:36.366 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:36.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.368 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:36.413 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:36.413 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:36.621 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000011.57.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000004.59.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000008.24.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:36.622 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:36.622 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.623 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.623 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:36.666 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:36.666 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:36.877 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.57.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.59.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:36.877 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:36.878 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:36.879 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:36.962 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:36.962 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:37.187 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000011.57.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000004.59.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000008.24.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:37.188 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:37.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:37.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:37.190 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:37.270 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:37.271 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:37.472 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.57.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.59.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:37.591 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:37.592 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:37.592 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:37.593 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:37.641 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:37.641 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:37.893 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.57.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.24.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:37.948 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-17 2021-03-02T12:09:37.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:37.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:37.950 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:37.992 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:37.992 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:38.191 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.53.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.51.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:38.312 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:38.314 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:38.314 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:38.315 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:38.360 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:38.360 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:38.575 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000004.53.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000011.51.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000009.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:38.694 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:38.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:38.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:38.931 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:38.975 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:38.976 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:39.190 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.53.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.51.5"},{"key":4,"val":""},{"key":5,"val":"00000000009.29.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:39.190 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:39.191 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:39.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:39.192 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:39.237 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:39.238 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:39.467 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000004.53.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000011.51.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000009.29.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:39.508 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-17 2021-03-02T12:09:39.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:39.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:39.510 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:39.551 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:39.739 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.33.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.35.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:39.894 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:39.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:39.895 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:39.895 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:39.944 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:39.945 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:40.164 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000004.33.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000004.35.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000015.38.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:40.164 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:40.165 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:40.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:40.166 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:40.251 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:40.251 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:40.460 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.33.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.35.5"},{"key":4,"val":""},{"key":5,"val":"00000000015.38.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:40.492 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:40.492 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:40.493 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:40.493 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:40.539 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:40.539 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:40.743 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"ESt8W4EqX2x-oE1BxH70uhycvObvG.X","ns":""},"count":1},"inc_marker":{"position":"00000000004.33.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"D5IirBCbTveVAmkEPTaP-01ItQfp6BQ","ns":""},"count":1},"inc_marker":{"position":"00000000004.35.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"KdvEsRDdQyrNcUItB.zeLJC0mRSUIAU","ns":""},"count":1},"inc_marker":{"position":"00000000015.38.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:40.744 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-17 2021-03-02T12:09:40.744 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-17 zones={test-zone2, test-zone1} 2021-03-02T12:09:40.755 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:40.760 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:40.760 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:40.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:40.761 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:40.761 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:40.772 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:40.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:40.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:40.773 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:40.787 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:40.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:40.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:40.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:41.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:41.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:41.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:41.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:41.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:41.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:41.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:42.041 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:42.174 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-17 zones={test-zone2, test-zone1} 2021-03-02T12:09:42.176 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-17 zones={test-zone3, test-zone1} 2021-03-02T12:09:42.197 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:42.210 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:42.211 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:42.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:42.212 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:42.213 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:42.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:42.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:42.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:42.229 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:42.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:42.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:42.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:42.473 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:42.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:42.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:42.813 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:42.879 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:43.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:43.115 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:43.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:43.287 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:43.396 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-17 zones={test-zone3, test-zone1} 2021-03-02T12:09:43.397 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-17 zones={test-zone3, test-zone2} 2021-03-02T12:09:43.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:43.419 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:43.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:43.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:43.420 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:43.421 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:43.427 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:43.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:43.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:43.428 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:43.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:43.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:43.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:43.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:43.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:43.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:43.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:43.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:44.034 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:44.145 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:44.202 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:44.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:44.430 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-17 zones={test-zone3, test-zone2} 2021-03-02T12:09:44.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:44.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:44.434 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:44.478 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:44.479 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:44.691 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.64.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:44.692 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:44.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:44.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:44.693 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:44.737 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:44.737 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:44.955 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000011.64.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000008.26.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:44.956 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:44.956 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:44.957 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:44.957 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:44.998 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:45.206 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.64.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:45.207 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:45.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:45.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:45.208 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:45.295 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:45.296 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:45.514 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000011.64.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000008.26.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:45.515 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:45.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:45.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:45.519 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:45.561 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:45.562 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:45.745 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.64.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.18.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.26.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:45.746 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:45.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:45.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:45.748 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:45.794 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:45.795 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:46.056 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.64.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.18.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.26.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:46.057 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-18 2021-03-02T12:09:46.058 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.059 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:46.141 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:46.141 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:46.349 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.31.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:46.350 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:46.351 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.352 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:46.398 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:46.399 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:46.627 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000008.24.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:46.628 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:46.629 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.630 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.631 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:46.675 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:46.676 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:46.893 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.31.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.31.5"},{"key":4,"val":""},{"key":5,"val":"00000000008.24.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:46.893 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:46.894 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.894 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:46.895 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:46.940 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:46.941 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:47.162 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000011.31.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000008.24.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:47.162 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-18 2021-03-02T12:09:47.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.164 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:47.204 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:47.204 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:47.400 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000016.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:47.400 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:47.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.402 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:47.485 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:47.486 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:47.695 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000004.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000016.40.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:47.695 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:47.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.697 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:47.778 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:47.779 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:47.989 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.29.5"},{"key":2,"val":""},{"key":3,"val":"00000000004.17.5"},{"key":4,"val":""},{"key":5,"val":"00000000016.40.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:47.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:47.990 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:47.991 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:48.037 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:48.037 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:48.254 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"en32fF.a37Jm7k3xNix9t6uMWAANV0O","ns":""},"count":1},"inc_marker":{"position":"00000000004.29.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"-oJR56Kp-MyZ29eACP8ZvHkIcKeX47O","ns":""},"count":1},"inc_marker":{"position":"00000000004.17.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"3qKoPc8GleSRdkHuFnif63wcHqpj2M8","ns":""},"count":1},"inc_marker":{"position":"00000000016.40.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:48.255 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-18 2021-03-02T12:09:48.256 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-18 zones={test-zone2, test-zone1} 2021-03-02T12:09:48.266 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:48.272 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:48.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:48.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:48.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:48.273 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:48.282 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:48.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:48.283 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:48.284 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:48.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:48.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:48.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:48.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:48.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:48.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:48.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:48.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:49.032 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:49.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:49.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:49.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:49.456 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-18 zones={test-zone2, test-zone1} 2021-03-02T12:09:49.457 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-18 zones={test-zone3, test-zone1} 2021-03-02T12:09:49.465 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:49.472 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:49.472 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:49.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:49.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:49.475 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:49.481 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:49.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:49.482 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:49.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:49.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:49.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:49.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:49.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:49.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:49.891 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:50.037 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:50.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:50.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:50.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:50.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:50.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:50.805 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-18 zones={test-zone3, test-zone1} 2021-03-02T12:09:50.952 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-18 zones={test-zone3, test-zone2} 2021-03-02T12:09:50.963 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:09:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:50.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:50.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:50.971 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:09:50.978 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:09:50.978 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-02T12:09:50.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-02T12:09:50.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone3 2021-03-02T12:09:50.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:51.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:51.081 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:09:51.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:51.208 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:51.330 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-02T12:09:51.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:51.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:51.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-02T12:09:51.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:51.960 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:52.095 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone3 2021-03-02T12:09:52.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-18 zones={test-zone3, test-zone2} 2021-03-02T12:09:52.211 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-03-02T12:09:52.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:52.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:52.259 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:52.298 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:52.298 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:52.475 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:09:52.507 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614686992.254509_152.1","last_update":"2021-03-02T12:09:52.254509Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:09:52.508 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:09:52.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:52.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:52.509 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:52.554 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:52.555 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:52.743 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686992.254509_152.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:52.254509Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:09:52.744 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:09:52.745 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:09:52.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:52.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:52.747 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:52.793 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:52.998 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686992.254509_152.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:52.254509Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:09:53.001 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:09:53.002 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:09:53.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:53.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:53.005 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:53.047 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:53.049 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:53.314 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686992.254509_152.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:52.254509Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:09:53.315 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:09:53.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:53.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:53.378 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:53.422 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:53.636 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.17.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:53.637 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:09:53.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:53.639 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:53.640 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:53.685 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:53.686 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:53.900 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:53.901 WARNING:rgw_multi.tests:bucket majrnq-19 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.17.10 target= 2021-03-02T12:09:58.902 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:58.983 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:58.984 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:59.029 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:59.030 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:59.250 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:59.251 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:09:59.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:59.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:59.254 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:59.297 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:59.515 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:09:59.516 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:09:59.516 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:59.517 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:59.517 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:59.606 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:59.607 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:09:59.825 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:09:59.850 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:09:59.851 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:59.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:09:59.852 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:09:59.893 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:09:59.894 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:00.099 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:00.152 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:00.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:00.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:00.153 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:00.199 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:00.199 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:00.567 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:00.775 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:00.775 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:00.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:00.776 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:00.818 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:00.819 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:01.041 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.18.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:01.188 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:01.189 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.190 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.191 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:01.242 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:01.243 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:01.464 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:01.464 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:01.465 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.465 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.466 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:01.507 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:01.721 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:01.723 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.18.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:01.724 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:01.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.726 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:01.770 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:01.771 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:01.988 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:01.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:01.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.990 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:01.991 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:02.070 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:02.278 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:02.279 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:02.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:02.280 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:02.280 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:02.363 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:02.363 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:02.593 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:02.594 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:02.595 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:02.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:02.597 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:02.680 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:02.681 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:02.878 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:02.879 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:02.880 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:02.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:02.882 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:02.929 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:02.930 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:03.159 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:03.160 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:03.160 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-19 zones={test-zone2, test-zone1} 2021-03-02T12:10:03.177 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:10:03.228 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:03.229 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:10:03.242 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:03.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.485 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-19 zones={test-zone2, test-zone1} 2021-03-02T12:10:03.486 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-19 zones={test-zone3, test-zone1} 2021-03-02T12:10:03.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:10:03.556 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:03.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:10:03.566 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:03.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.820 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-19 zones={test-zone3, test-zone1} 2021-03-02T12:10:03.821 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-19 zones={test-zone3, test-zone2} 2021-03-02T12:10:03.836 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:10:03.897 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:03.898 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:10:03.910 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:03.976 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:03.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:04.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:04.158 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-19 zones={test-zone3, test-zone2} 2021-03-02T12:10:06.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.049 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:06.092 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:06.093 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:06.306 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000044.147.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:06.307 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:06.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.309 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:06.357 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:06.357 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:06.554 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"j3eHIBgZl33NxmOq096TJgchRxt9b4b","ns":""},"count":1},"inc_marker":{"position":"00000000044.147.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:06.555 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:06.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.557 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.558 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:06.602 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:06.602 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:06.794 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000050.160.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:06.795 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:06.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:06.797 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:06.877 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:06.878 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:07.098 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"j3eHIBgZl33NxmOq096TJgchRxt9b4b","ns":""},"count":2},"inc_marker":{"position":"00000000044.147.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:07.099 WARNING:rgw_multi.tests:bucket majrnq-19 zone test-zone3 behind zone test-zone1: shard=8 master=00000000050.160.9 target=00000000044.147.11 2021-03-02T12:10:12.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:12.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:12.104 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:12.148 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:12.149 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:12.392 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"j3eHIBgZl33NxmOq096TJgchRxt9b4b","ns":""},"count":2},"inc_marker":{"position":"00000000051.162.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:12.475 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:12.475 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:12.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:12.477 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:12.518 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:12.567 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:12.721 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000005.23.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000051.162.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:12.721 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:12.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:12.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:12.723 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:12.770 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:12.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:13.035 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.23.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000051.162.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:13.035 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-19 2021-03-02T12:10:13.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.037 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:13.078 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:13.079 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:13.277 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.18.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000049.157.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:13.278 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:13.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.280 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:13.325 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:13.556 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nQZdU3A-ujVI.g1anMtFpYiCphDRBMW","ns":""},"count":15},"inc_marker":{"position":"00000000049.157.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:13.557 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:13.557 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.558 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:13.607 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:13.608 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:13.817 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.18.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000049.157.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:13.818 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:13.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:13.819 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:13.865 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:13.866 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:14.106 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.18.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"nQZdU3A-ujVI.g1anMtFpYiCphDRBMW","ns":""},"count":15},"inc_marker":{"position":"00000000049.157.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:14.107 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-19 2021-03-02T12:10:14.108 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:14.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:14.109 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:14.193 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:14.193 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:14.413 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.147.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:14.413 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:14.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:14.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:14.415 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:14.501 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:14.501 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:14.733 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"j3eHIBgZl33NxmOq096TJgchRxt9b4b","ns":""},"count":4},"inc_marker":{"position":"00000000047.147.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:14.733 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:14.734 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:14.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-19', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:14.735 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-19 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:14.825 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:14.825 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:15.029 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000004.19.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000047.147.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:15.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:15.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:15.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-19', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:15.031 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-19 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:15.082 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:15.083 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:15.294 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"MuQbYNll.bzKLR2sHortT3x5CAFtj7d","ns":""},"count":1},"inc_marker":{"position":"00000000004.19.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"j3eHIBgZl33NxmOq096TJgchRxt9b4b","ns":""},"count":4},"inc_marker":{"position":"00000000047.147.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:15.295 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-19 2021-03-02T12:10:15.295 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-19 zones={test-zone2, test-zone1} 2021-03-02T12:10:15.308 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:10:15.320 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:15.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.321 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.322 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.324 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.325 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.326 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.327 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.328 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.329 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.330 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.331 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.332 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.333 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.334 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:10:15.350 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:15.350 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.355 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.356 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.364 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:15.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:15.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:15.487 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:15.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:15.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:15.734 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:15.843 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:15.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.419 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:16.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.679 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:17.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.661 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:18.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.120 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.416 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:19.952 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.520 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:20.944 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.477 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:21.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.019 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:22.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.464 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.826 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:23.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:24.857 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-19 zones={test-zone2, test-zone1} 2021-03-02T12:10:24.858 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-19 zones={test-zone3, test-zone1} 2021-03-02T12:10:24.874 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:10:24.892 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:24.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.895 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.896 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.897 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.898 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:10:24.918 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:24.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.919 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.920 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.921 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.922 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.924 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.930 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.931 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:24.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:24.981 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:25.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:25.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:25.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:25.405 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:25.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:25.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:25.812 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:25.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.559 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.619 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:26.903 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.338 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:27.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.643 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.830 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:28.996 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.519 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:29.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.108 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.455 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:30.993 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.284 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.468 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:31.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:32.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:32.271 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:32.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:32.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:32.818 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:32.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.004 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:33.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:34.177 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:34.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:34.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:34.609 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:34.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:34.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:35.089 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:35.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:35.518 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-19 zones={test-zone3, test-zone1} 2021-03-02T12:10:35.519 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-19 zones={test-zone3, test-zone2} 2021-03-02T12:10:35.545 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:10:35.570 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:35.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.573 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.574 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.575 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.576 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.577 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.579 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.580 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.581 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.582 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.584 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:10:35.605 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-02T12:10:35.605 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.606 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.608 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.618 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.619 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.620 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.621 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:10:35.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:35.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:35.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-02T12:10:35.885 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:35.968 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.131 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:36.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.234 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.286 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.523 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.819 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:37.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.183 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.444 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:38.935 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.339 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.395 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.825 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.939 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:39.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.125 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.587 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.710 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.820 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:40.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:41.893 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.006 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.059 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.899 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:42.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.112 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:43.959 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.401 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.515 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:44.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:45.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:10:45.112 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-19 zones={test-zone3, test-zone2} 2021-03-02T12:10:45.113 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.114 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:45.154 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:45.155 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:45.362 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687018.113106_75.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"1_1614687018.113106_165.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686974.105927_93.3","last_update":"2021-03-02T12:09:34.105927Z"}] 2021-03-02T12:10:45.362 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:10:45.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.364 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:45.407 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:45.408 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:45.601 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:45.602 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:10:45.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.604 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:45.660 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:45.660 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:45.863 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687018.113106_75.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"1_1614687018.113106_165.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686974.105927_93.3","last_update":"2021-03-02T12:09:34.105927Z"}] 2021-03-02T12:10:45.864 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:10:45.865 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:45.866 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:45.953 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:45.954 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:46.157 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7196637147885404157},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:46.158 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:10:46.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:46.159 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:46.159 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:46.241 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:46.242 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:46.448 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687018.113106_75.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"1_1614687018.113106_165.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686974.105927_93.3","last_update":"2021-03-02T12:09:34.105927Z"}] 2021-03-02T12:10:46.448 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:10:46.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:46.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:46.450 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:46.494 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:46.494 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:46.763 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_75.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:46.764 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:10:46.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:46.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:46.765 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:46.849 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:46.849 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:47.039 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687019.051884_71.1","last_update":"2021-03-02T12:10:19.051884Z"},{"marker":"1_1614686997.049307_184.1","last_update":"2021-03-02T12:09:57.049307Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686975.042174_98.3","last_update":"2021-03-02T12:09:35.042174Z"}] 2021-03-02T12:10:47.040 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:10:47.042 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.043 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.043 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:47.088 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:47.286 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687019.051884_71.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:19.051884Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:47.287 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:10:47.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.288 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:47.329 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:47.329 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:47.513 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687019.051884_71.1","last_update":"2021-03-02T12:10:19.051884Z"},{"marker":"1_1614686997.049307_184.1","last_update":"2021-03-02T12:09:57.049307Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686975.042174_98.3","last_update":"2021-03-02T12:09:35.042174Z"}] 2021-03-02T12:10:47.513 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:10:47.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.514 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.515 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:47.561 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:47.561 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:47.766 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13674358676509122085},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687019.051884_71.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:19.051884Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:47.767 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:10:47.767 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.768 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:47.768 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:47.848 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:47.848 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:48.062 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687018.101464_31.1","last_update":"2021-03-02T12:10:18.101464Z"},{"marker":"1_1614686996.099228_141.1","last_update":"2021-03-02T12:09:56.099228Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686974.092078_69.3","last_update":"2021-03-02T12:09:34.092078Z"}] 2021-03-02T12:10:48.063 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:10:48.063 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.064 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.064 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:48.146 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:48.147 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:48.368 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14630138824005222133},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687018.101464_31.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.101464Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686996.099228_141.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:56.099228Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.092078_69.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.092078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:48.369 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:10:48.369 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.370 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:48.452 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:48.452 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:48.666 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687018.101464_31.1","last_update":"2021-03-02T12:10:18.101464Z"},{"marker":"1_1614686996.099228_141.1","last_update":"2021-03-02T12:09:56.099228Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1614686974.092078_69.3","last_update":"2021-03-02T12:09:34.092078Z"}] 2021-03-02T12:10:48.667 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:10:48.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.668 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:48.713 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:48.713 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:48.906 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9850944892582908325},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687018.101464_31.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.101464Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686996.099228_141.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:56.099228Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.092078_69.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.092078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:10:48.907 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:10:48.908 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-03-02T12:10:48.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:48.945 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:48.983 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:48.983 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:49.156 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:10:49.189 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687048.939900_172.1","last_update":"2021-03-02T12:10:48.939900Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:10:49.190 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:10:49.190 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:49.191 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:49.191 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:49.229 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:49.229 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:49.422 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614686992.254509_152.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:52.254509Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:10:49.423 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687048.939900_172.1 target=1_1614686992.254509_152.1 2021-03-02T12:10:54.424 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:54.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:54.425 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:54.466 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:54.466 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:54.660 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687048.939900_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:48.939900Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:10:54.660 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:10:54.661 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:10:54.661 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:54.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:54.662 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:54.706 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:54.706 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:54.908 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687048.939900_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:48.939900Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:10:54.916 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:10:54.917 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:10:54.917 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:54.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:54.918 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:54.960 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:54.960 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:55.224 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687048.939900_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:48.939900Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:10:55.224 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:10:55.279 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:55.279 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:55.280 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:55.322 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:55.398 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:55.537 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:10:55.538 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:10:55.538 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:55.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:10:55.539 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:10:55.584 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:10:55.692 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:10:55.785 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:10:55.785 WARNING:rgw_multi.tests:bucket majrnq-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000001.28.5 target= 2021-03-02T12:11:00.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:00.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:00.791 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:00.837 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:00.838 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:01.042 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:01.043 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:01.043 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.044 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.044 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:01.086 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:01.319 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:01.320 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:01.321 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.322 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:01.407 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:01.408 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:01.634 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:01.635 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:01.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.636 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:01.678 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:01.678 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:01.891 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.28.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:01.892 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:01.892 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.893 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:01.893 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:01.938 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:01.938 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:02.303 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:02.303 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:02.304 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:02.304 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:02.305 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:02.386 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:02.387 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:02.579 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:02.579 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:02.580 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:02.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:02.581 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:02.626 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:02.627 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:02.869 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:02.870 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:02.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:02.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:02.871 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:02.915 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:02.916 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:03.108 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.23.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:03.109 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:03.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:03.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:03.110 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:03.157 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:03.383 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:03.383 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:03.384 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:03.384 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:03.385 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:03.467 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:03.474 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:03.673 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:03.674 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:03.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:03.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:03.676 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:03.757 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:03.970 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:04.005 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:04.006 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:04.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:04.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:04.008 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:04.089 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:04.272 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:04.295 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:04.295 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:04.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:04.296 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:04.297 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:04.341 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:04.342 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:04.546 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:04.547 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:04.548 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone2, test-zone1} 2021-03-02T12:11:04.560 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:04.607 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:04.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:04.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:04.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:04.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:04.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:04.969 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone2, test-zone1} 2021-03-02T12:11:05.321 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone3, test-zone1} 2021-03-02T12:11:05.337 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:05.387 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:05.387 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:05.396 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:05.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:05.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:05.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:05.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone3, test-zone1} 2021-03-02T12:11:05.656 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone3, test-zone2} 2021-03-02T12:11:05.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:05.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:05.715 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:05.722 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:05.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:05.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:05.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:05.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone3, test-zone2} 2021-03-02T12:11:06.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:06.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:06.122 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:06.162 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:06.163 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:06.337 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:11:06.370 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687066.118540_180.1","last_update":"2021-03-02T12:11:06.118540Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:11:06.371 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:11:06.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:06.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:06.373 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:06.415 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:06.415 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:06.614 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687048.939900_172.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:48.939900Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:06.615 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687066.118540_180.1 target=1_1614687048.939900_172.1 2021-03-02T12:11:11.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:11.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:11.620 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:11.661 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:11.661 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:11.846 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687066.118540_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:06.118540Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:11.846 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:11:11.847 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:11:11.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:11.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:11.849 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:11.890 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:11.890 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:12.097 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687066.118540_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:06.118540Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:12.098 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:11:12.098 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:11:12.099 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:12.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:12.100 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:12.141 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:12.142 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:12.402 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687066.118540_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:06.118540Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:12.403 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:11:12.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:12.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:12.456 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:12.496 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:12.496 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:12.713 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:12.714 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:12.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:12.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:12.715 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:12.760 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:12.760 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:12.972 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.28.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:12.972 WARNING:rgw_multi.tests:bucket majrnq-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.31.12 target=00000000001.28.5 2021-03-02T12:11:17.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:17.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:17.980 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:18.025 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:18.026 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:18.234 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:18.235 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:18.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:18.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:18.237 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:18.281 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:18.282 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:18.489 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:18.490 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:18.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:18.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:18.492 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:18.585 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:18.586 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:18.819 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:18.819 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:18.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:18.821 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:18.821 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:18.907 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:18.908 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:19.120 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:19.120 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:19.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.122 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:19.170 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:19.171 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:19.435 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:19.436 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:19.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.439 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:19.519 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:19.519 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:19.707 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.30.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:19.708 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:19.710 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.711 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:19.758 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:19.758 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:19.978 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.30.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:19.979 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:19.980 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.980 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:19.981 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:20.023 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:20.023 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:20.218 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.30.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:20.219 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:20.220 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:20.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:20.221 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:20.267 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:20.267 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:20.489 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.30.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:20.490 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:20.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:20.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:20.491 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:20.570 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:20.571 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:20.774 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.31.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:20.774 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:20.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:20.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:20.776 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:20.860 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:21.094 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:21.094 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:21.095 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:21.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:21.096 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:21.137 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:21.342 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000004.31.9"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:21.343 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:21.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:21.344 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:21.344 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:21.390 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:21.391 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:21.601 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.31.9"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:21.601 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:21.602 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone2, test-zone1} 2021-03-02T12:11:21.613 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:21.663 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:21.664 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:21.664 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:21.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:21.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:21.732 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:21.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:21.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:21.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:21.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:21.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone2, test-zone1} 2021-03-02T12:11:22.020 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone3, test-zone1} 2021-03-02T12:11:22.029 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:22.079 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.081 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.081 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:22.089 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.091 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.148 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.434 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone3, test-zone1} 2021-03-02T12:11:22.435 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone3, test-zone2} 2021-03-02T12:11:22.445 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:22.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.496 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:22.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.503 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:22.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.572 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.709 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:22.853 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone3, test-zone2} 2021-03-02T12:11:22.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:22.880 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:22.880 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:22.920 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:22.920 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:23.095 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:11:23.130 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687082.865578_188.1","last_update":"2021-03-02T12:11:22.865578Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:11:23.130 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:11:23.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:23.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:23.149 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:23.191 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:23.192 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:23.386 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687066.118540_180.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:06.118540Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:23.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687082.865578_188.1 target=1_1614687066.118540_180.1 2021-03-02T12:11:28.390 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:28.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:28.410 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:28.466 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:28.466 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:28.660 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687082.865578_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:22.865578Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:28.661 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:11:28.662 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:11:28.663 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:28.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:28.665 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:28.726 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:28.727 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:28.934 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687082.865578_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:22.865578Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:28.937 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:11:28.942 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:11:28.945 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:28.949 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:28.952 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:28.999 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:29.000 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:29.265 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687082.865578_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:22.865578Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:29.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:11:29.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:29.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:29.324 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:29.367 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:29.560 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.35.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:29.561 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:29.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:29.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:29.573 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:29.634 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:29.634 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:29.837 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:29.838 WARNING:rgw_multi.tests:bucket majrnq-20 zone test-zone2 behind zone test-zone1: shard=8 master=00000000005.35.17 target=00000000003.31.12 2021-03-02T12:11:34.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:34.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:34.863 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:34.910 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:34.911 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:35.128 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.35.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:35.128 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:35.129 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:35.129 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:35.130 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:35.174 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:35.425 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:35.426 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.35.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:35.427 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:35.427 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:35.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:35.428 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:35.514 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:35.515 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:35.721 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.35.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:35.811 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:35.812 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:35.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:35.813 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:35.857 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:35.858 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:36.117 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.35.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:36.119 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:36.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.120 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:36.165 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:36.165 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:36.429 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.35.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:36.429 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-20 2021-03-02T12:11:36.432 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.433 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:36.474 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:36.475 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:36.668 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.38.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:36.688 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:36.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.690 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:36.736 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:36.736 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:36.978 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.38.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:36.978 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:36.979 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.979 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:36.980 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:37.022 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:37.023 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:37.228 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.38.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:37.228 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:37.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:37.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:37.230 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:37.273 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:37.432 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:37.505 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.38.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:37.506 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-20 2021-03-02T12:11:37.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:37.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:37.508 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:37.593 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:37.650 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:37.785 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.39.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:37.833 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:37.834 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:37.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:37.835 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:37.881 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:38.132 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.39.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:38.150 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:38.151 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:38.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-20', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:38.152 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-20 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:38.219 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:38.436 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000007.39.14"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:38.437 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:38.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:38.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-20', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:38.439 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-20 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:38.483 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:38.483 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:38.690 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000007.39.14"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:38.691 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-20 2021-03-02T12:11:38.692 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone2, test-zone1} 2021-03-02T12:11:38.726 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:38.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:38.815 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:38.816 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:38.841 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:38.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:38.949 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.530 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone2, test-zone1} 2021-03-02T12:11:39.531 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone3, test-zone1} 2021-03-02T12:11:39.545 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:39.600 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:39.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:39.601 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:39.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:39.610 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:39.676 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.782 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:39.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone3, test-zone1} 2021-03-02T12:11:40.020 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-20 zones={test-zone3, test-zone2} 2021-03-02T12:11:40.037 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:40.092 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:40.093 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:40.094 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:40.103 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:40.104 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:40.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.188 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.367 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.449 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:40.528 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-20 zones={test-zone3, test-zone2} 2021-03-02T12:11:40.528 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-03-02T12:11:40.531 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-21 2021-03-02T12:11:40.570 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-22 2021-03-02T12:11:40.630 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-23 2021-03-02T12:11:40.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:40.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:40.736 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:40.780 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:40.781 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:40.964 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:11:40.999 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687100.726538_212.1","last_update":"2021-03-02T12:11:40.726538Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:11:41.000 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:11:41.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:41.001 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:41.002 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:41.045 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:41.046 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:41.239 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687082.865578_188.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:22.865578Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:41.240 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687100.726538_212.1 target=1_1614687082.865578_188.1 2021-03-02T12:11:46.242 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:46.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:46.244 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:46.286 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:46.286 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:46.476 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687100.726538_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:40.726538Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:46.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:11:46.478 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:11:46.479 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:46.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:46.481 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:46.523 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:46.523 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:46.735 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687100.726538_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:40.726538Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:46.904 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:11:46.906 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:11:46.907 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:46.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:46.908 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:46.948 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:46.948 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:47.215 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687100.726538_212.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:11:40.726538Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:11:47.215 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:11:47.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:47.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:47.716 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:47.756 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:47.980 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.13.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:48.311 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-21 2021-03-02T12:11:48.312 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:48.312 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:48.313 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:48.357 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:48.357 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:48.537 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RpzhCerfKyPjLKGnklcrWP1i7c8ZumP","ns":""},"count":2},"inc_marker":{"position":"00000000003.13.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:48.808 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-21 2021-03-02T12:11:48.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:48.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:48.810 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:48.859 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:48.860 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:49.079 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.13.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:49.079 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-21 2021-03-02T12:11:49.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:49.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:49.082 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:49.172 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:49.173 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:49.396 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RpzhCerfKyPjLKGnklcrWP1i7c8ZumP","ns":""},"count":2},"inc_marker":{"position":"00000000003.13.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:49.397 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-21 2021-03-02T12:11:49.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:49.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:49.399 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:49.513 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:49.513 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:49.720 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.13.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:49.720 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-21 2021-03-02T12:11:49.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:49.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:49.722 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:49.774 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:49.774 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:50.041 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.13.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:50.041 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-21 2021-03-02T12:11:50.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.043 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:50.127 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:50.128 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:50.329 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:50.330 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-21 2021-03-02T12:11:50.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.331 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:50.380 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:50.582 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RpzhCerfKyPjLKGnklcrWP1i7c8ZumP","ns":""},"count":2},"inc_marker":{"position":"00000000002.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:50.583 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-21 2021-03-02T12:11:50.584 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.584 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.585 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:50.631 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:50.634 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:50.812 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.13.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:50.813 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-21 2021-03-02T12:11:50.814 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:50.816 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:50.865 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:50.865 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:51.118 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RpzhCerfKyPjLKGnklcrWP1i7c8ZumP","ns":""},"count":2},"inc_marker":{"position":"00000000002.13.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:51.119 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-21 2021-03-02T12:11:51.120 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:51.120 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:51.121 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:51.203 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:51.414 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:51.415 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-21 2021-03-02T12:11:51.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:51.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:51.416 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:51.502 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:51.503 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:51.721 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RpzhCerfKyPjLKGnklcrWP1i7c8ZumP","ns":""},"count":2},"inc_marker":{"position":"00000000002.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:51.722 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-21 2021-03-02T12:11:51.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:51.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-21', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:51.724 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-21 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:51.803 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:51.804 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:52.031 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:52.031 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-21 2021-03-02T12:11:52.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:52.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-21', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:52.033 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-21 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:52.079 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:52.080 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:52.292 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"RpzhCerfKyPjLKGnklcrWP1i7c8ZumP","ns":""},"count":2},"inc_marker":{"position":"00000000002.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:52.293 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-21 2021-03-02T12:11:52.293 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-21 zones={test-zone2, test-zone1} 2021-03-02T12:11:52.304 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:52.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.351 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.352 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:52.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.437 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.506 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-21 zones={test-zone2, test-zone1} 2021-03-02T12:11:52.506 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-21 zones={test-zone3, test-zone1} 2021-03-02T12:11:52.516 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.563 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.564 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:52.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.572 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.719 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-21 zones={test-zone3, test-zone1} 2021-03-02T12:11:52.720 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-21 zones={test-zone3, test-zone2} 2021-03-02T12:11:52.733 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:52.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.787 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.788 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:52.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:52.853 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.862 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.872 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:52.940 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-21 zones={test-zone3, test-zone2} 2021-03-02T12:11:52.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:52.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:52.942 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:52.982 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:52.982 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:53.199 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:53.199 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-22 2021-03-02T12:11:53.200 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:53.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:53.201 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:53.246 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:53.247 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:53.457 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zE5Utc7WHHWOcoZTTM-kSwdHWGCEoNe","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:53.457 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-22 2021-03-02T12:11:53.458 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:53.459 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:53.459 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:53.501 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:53.502 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:53.712 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:53.713 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-22 2021-03-02T12:11:53.713 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:53.714 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:53.715 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:53.799 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:53.799 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:54.055 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zE5Utc7WHHWOcoZTTM-kSwdHWGCEoNe","ns":""},"count":1},"inc_marker":{"position":"00000000002.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:54.056 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-22 2021-03-02T12:11:54.056 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.057 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:54.099 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:54.313 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.24.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:54.315 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-22 2021-03-02T12:11:54.317 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.318 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.320 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:54.379 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:54.380 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:54.645 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.24.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:54.646 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-22 2021-03-02T12:11:54.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.647 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:54.737 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:54.738 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:54.936 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:54.937 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-22 2021-03-02T12:11:54.938 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:54.939 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:54.986 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:54.986 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:55.234 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zE5Utc7WHHWOcoZTTM-kSwdHWGCEoNe","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:55.234 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-22 2021-03-02T12:11:55.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:55.235 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:55.236 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:55.279 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:55.279 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:55.490 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.28.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:55.491 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-22 2021-03-02T12:11:55.491 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:55.492 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:55.492 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:55.543 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:55.544 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:55.762 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zE5Utc7WHHWOcoZTTM-kSwdHWGCEoNe","ns":""},"count":1},"inc_marker":{"position":"00000000003.28.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:55.763 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-22 2021-03-02T12:11:55.763 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:55.764 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:55.764 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:55.852 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:55.853 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:56.090 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.31.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:56.090 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-22 2021-03-02T12:11:56.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:56.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:56.092 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:56.179 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:56.180 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:56.430 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zE5Utc7WHHWOcoZTTM-kSwdHWGCEoNe","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:56.430 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-22 2021-03-02T12:11:56.431 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:56.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-22', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:56.432 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-22 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:56.513 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:56.514 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:56.727 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.31.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:56.727 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-22 2021-03-02T12:11:56.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:56.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-22', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:56.729 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-22 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:56.775 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:56.775 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:56.997 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"zE5Utc7WHHWOcoZTTM-kSwdHWGCEoNe","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:56.998 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-22 2021-03-02T12:11:56.999 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-22 zones={test-zone2, test-zone1} 2021-03-02T12:11:57.009 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:57.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.055 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.056 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:57.064 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.065 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.220 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-22 zones={test-zone2, test-zone1} 2021-03-02T12:11:57.220 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-22 zones={test-zone3, test-zone1} 2021-03-02T12:11:57.230 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:57.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.279 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.280 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:57.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.288 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.439 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-22 zones={test-zone3, test-zone1} 2021-03-02T12:11:57.440 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-22 zones={test-zone3, test-zone2} 2021-03-02T12:11:57.449 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:11:57.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.499 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.500 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:11:57.506 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:11:57.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:11:57.655 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-22 zones={test-zone3, test-zone2} 2021-03-02T12:11:57.655 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:57.656 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:57.657 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:57.700 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:57.917 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.42.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:57.917 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-23 2021-03-02T12:11:57.918 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:57.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:57.919 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:57.962 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:57.963 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:58.184 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Ofjp9swMoT92H1eNTphyZFjZtJdk52b","ns":""},"count":1},"inc_marker":{"position":"00000000002.42.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:58.185 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-23 2021-03-02T12:11:58.185 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:58.186 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:58.186 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:58.228 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:58.229 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:58.455 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.42.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:58.455 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-23 2021-03-02T12:11:58.456 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:58.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:58.457 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:58.541 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:58.767 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Ofjp9swMoT92H1eNTphyZFjZtJdk52b","ns":""},"count":1},"inc_marker":{"position":"00000000002.42.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:58.768 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-23 2021-03-02T12:11:58.769 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:58.769 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:58.770 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:58.852 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:58.853 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:59.076 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.42.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:59.077 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-23 2021-03-02T12:11:59.077 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:59.078 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:59.078 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:59.123 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:59.124 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:59.390 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:11:59.391 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-23 2021-03-02T12:11:59.391 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:59.392 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:59.392 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:59.472 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:59.472 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:59.674 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.47.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:11:59.674 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-23 2021-03-02T12:11:59.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:59.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:11:59.676 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:11:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:11:59.722 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:11:59.976 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Ofjp9swMoT92H1eNTphyZFjZtJdk52b","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:00.006 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-23 2021-03-02T12:12:00.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.008 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:00.052 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:00.052 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:00.268 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.47.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:00.273 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-23 2021-03-02T12:12:00.274 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.275 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:00.320 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:00.321 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:00.550 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Ofjp9swMoT92H1eNTphyZFjZtJdk52b","ns":""},"count":1},"inc_marker":{"position":"00000000002.47.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:00.551 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-23 2021-03-02T12:12:00.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.552 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:00.597 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:00.597 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:00.808 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.51.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:00.908 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-23 2021-03-02T12:12:00.908 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:00.909 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:00.955 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:00.955 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:01.200 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Ofjp9swMoT92H1eNTphyZFjZtJdk52b","ns":""},"count":1},"inc_marker":{"position":"00000000003.51.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:01.201 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-23 2021-03-02T12:12:01.202 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:01.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-23', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:01.205 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-23 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:01.286 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:01.391 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:01.519 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.51.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:01.520 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-23 2021-03-02T12:12:01.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:01.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-23', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:01.521 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-23 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:01.566 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:01.566 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:01.774 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"Ofjp9swMoT92H1eNTphyZFjZtJdk52b","ns":""},"count":1},"inc_marker":{"position":"00000000003.51.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:01.775 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-23 2021-03-02T12:12:01.775 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-23 zones={test-zone2, test-zone1} 2021-03-02T12:12:01.787 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:01.848 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:01.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:01.973 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:01.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:01.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:02.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:02.071 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:02.149 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-23 zones={test-zone2, test-zone1} 2021-03-02T12:12:02.225 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-23 zones={test-zone3, test-zone1} 2021-03-02T12:12:02.238 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:02.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.287 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.288 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:02.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.296 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:02.767 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:02.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:02.869 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-23 zones={test-zone3, test-zone1} 2021-03-02T12:12:02.870 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-23 zones={test-zone3, test-zone2} 2021-03-02T12:12:02.879 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:02.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.927 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.928 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:02.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.935 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:02.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:03.250 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:03.262 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:03.345 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-23 zones={test-zone3, test-zone2} 2021-03-02T12:12:03.608 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-03-02T12:12:03.610 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-24 2021-03-02T12:12:03.642 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-25 2021-03-02T12:12:03.702 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-26 2021-03-02T12:12:04.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:04.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:04.249 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:04.290 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:04.505 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:04.506 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:12:04.516 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687124.239450_252.1","last_update":"2021-03-02T12:12:04.239450Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:12:04.517 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:12:04.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:04.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:04.519 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:04.562 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:04.562 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:04.812 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687124.239450_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:04.239450Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:04.813 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:12:04.814 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:12:04.815 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:04.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:04.816 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:04.890 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:05.085 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687124.239450_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:04.239450Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:05.086 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:12:05.086 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:12:05.087 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:05.087 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:05.088 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:05.158 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:05.158 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:05.420 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687124.239450_252.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:04.239450Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:05.421 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:12:05.681 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:05.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:05.683 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:05.735 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:05.736 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:05.946 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.26.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:05.947 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-24 2021-03-02T12:12:05.948 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:05.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:05.949 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:05.999 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:05.999 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:06.233 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:06.234 WARNING:rgw_multi.tests:bucket majrnq-24 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.26.6 target= 2021-03-02T12:12:11.235 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:11.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:11.237 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:11.282 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:11.282 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:11.503 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.26.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:11.593 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-24 2021-03-02T12:12:11.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:11.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:11.595 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:11.632 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:11.830 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.26.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:12.053 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-24 2021-03-02T12:12:12.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.056 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:12.102 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:12.335 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.26.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:12.337 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-24 2021-03-02T12:12:12.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.338 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:12.422 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:12.485 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:12.628 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.26.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:12.646 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-24 2021-03-02T12:12:12.646 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.647 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:12.702 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:12.702 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:12.963 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.26.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:12.988 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-24 2021-03-02T12:12:12.989 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:12.990 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:13.046 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:13.206 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:13.271 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.37.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:13.414 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-24 2021-03-02T12:12:13.415 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:13.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:13.416 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:13.461 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:13.507 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:13.684 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:13.707 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-24 2021-03-02T12:12:13.708 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:13.708 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:13.709 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:13.751 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:13.752 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:13.964 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.37.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:13.965 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-24 2021-03-02T12:12:13.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:13.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:13.966 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:14.013 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:14.014 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:14.245 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:14.245 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-24 2021-03-02T12:12:14.246 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:14.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:14.248 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:14.296 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:14.296 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:14.502 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.22.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:14.503 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-24 2021-03-02T12:12:14.504 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:14.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:14.506 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:14.592 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:14.592 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:14.817 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:14.818 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-24 2021-03-02T12:12:14.818 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:14.819 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-24', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:14.819 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-24 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:14.867 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:14.868 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:15.078 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.22.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:15.079 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-24 2021-03-02T12:12:15.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:15.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-24', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:15.080 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-24 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:15.173 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:15.174 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:15.410 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.22.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:15.411 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-24 2021-03-02T12:12:15.411 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-24 zones={test-zone2, test-zone1} 2021-03-02T12:12:15.436 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:15.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:15.506 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:15.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:15.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:15.613 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-24 zones={test-zone2, test-zone1} 2021-03-02T12:12:15.614 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-24 zones={test-zone3, test-zone1} 2021-03-02T12:12:15.639 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:15.688 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:15.688 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:15.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:15.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:15.770 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-24 zones={test-zone3, test-zone1} 2021-03-02T12:12:15.771 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-24 zones={test-zone3, test-zone2} 2021-03-02T12:12:15.791 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:15.844 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:15.844 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:15.851 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:15.919 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:15.934 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-24 zones={test-zone3, test-zone2} 2021-03-02T12:12:15.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:15.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:15.938 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:15.991 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:15.991 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:16.205 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:16.205 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-25 2021-03-02T12:12:16.206 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:16.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:16.207 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:16.259 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:16.259 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:16.498 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:16.499 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-25 2021-03-02T12:12:16.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:16.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:16.501 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:16.547 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:16.752 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:16.753 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-25 2021-03-02T12:12:16.754 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:16.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:16.755 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:16.844 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:17.079 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:17.080 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-25 2021-03-02T12:12:17.080 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.081 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:17.162 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:17.369 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.19.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:17.369 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-25 2021-03-02T12:12:17.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.372 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:17.419 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:17.419 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:17.683 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.19.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:17.683 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-25 2021-03-02T12:12:17.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.687 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:17.766 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:17.766 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:17.989 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:17.990 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-25 2021-03-02T12:12:17.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:17.991 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:18.036 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:18.037 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:18.255 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:18.256 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-25 2021-03-02T12:12:18.257 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:18.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:18.258 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:18.300 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:18.301 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:18.517 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.27.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:18.518 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-25 2021-03-02T12:12:18.518 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:18.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:18.520 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:18.566 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:18.566 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:18.779 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.27.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:18.780 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-25 2021-03-02T12:12:18.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:18.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:18.783 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:18.863 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:18.864 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:19.090 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:19.091 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-25 2021-03-02T12:12:19.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:19.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:19.092 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:19.178 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:19.179 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:19.397 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:19.398 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-25 2021-03-02T12:12:19.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:19.399 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-25', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:19.399 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-25 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:19.481 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:19.699 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.16.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:19.700 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-25 2021-03-02T12:12:19.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:19.701 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-25', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:19.702 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-25 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:19.749 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:19.749 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:19.972 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.16.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:19.972 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-25 2021-03-02T12:12:19.973 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-25 zones={test-zone2, test-zone1} 2021-03-02T12:12:19.982 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:20.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:20.031 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:20.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:20.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:20.109 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-25 zones={test-zone2, test-zone1} 2021-03-02T12:12:20.110 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-25 zones={test-zone3, test-zone1} 2021-03-02T12:12:20.122 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:20.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:20.173 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:20.182 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:20.246 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:20.254 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-25 zones={test-zone3, test-zone1} 2021-03-02T12:12:20.255 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-25 zones={test-zone3, test-zone2} 2021-03-02T12:12:20.273 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:20.323 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:20.324 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:20.342 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:20.406 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:20.416 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-25 zones={test-zone3, test-zone2} 2021-03-02T12:12:20.417 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:20.417 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:20.418 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:20.468 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:20.684 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:20.685 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-26 2021-03-02T12:12:20.685 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:20.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:20.686 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:20.731 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:20.732 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:20.946 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:20.947 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-26 2021-03-02T12:12:20.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:20.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:20.949 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:20.990 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:20.991 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:21.206 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:21.206 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-26 2021-03-02T12:12:21.207 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:21.207 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:21.208 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:21.291 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:21.292 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:21.532 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:21.533 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-26 2021-03-02T12:12:21.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:21.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:21.535 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:21.576 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:21.744 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:21.776 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.31.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:21.776 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-26 2021-03-02T12:12:21.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:21.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:21.778 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:21.825 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:21.825 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:22.091 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.31.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:22.092 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-26 2021-03-02T12:12:22.092 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.093 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.093 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:22.174 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:22.175 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:22.397 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.33.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:22.397 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-26 2021-03-02T12:12:22.398 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.398 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.399 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:22.449 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:22.695 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:22.697 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:22.698 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-26 2021-03-02T12:12:22.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.700 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:22.742 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:22.742 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:22.943 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.33.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:22.943 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-26 2021-03-02T12:12:22.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.945 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:22.946 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:22.996 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:22.997 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:23.229 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:23.230 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-26 2021-03-02T12:12:23.230 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:23.231 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:23.231 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:23.312 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:23.313 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:23.522 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.56.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:23.732 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-26 2021-03-02T12:12:23.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:23.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:23.735 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:23.780 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:23.780 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:24.036 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.56.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:24.066 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-26 2021-03-02T12:12:24.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:24.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-26', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:24.068 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-26 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:24.120 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:24.120 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:24.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.56.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:12:24.442 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-26 2021-03-02T12:12:24.443 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:24.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-26', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:24.444 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-26 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:24.489 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:24.757 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:24.759 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.56.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:12:24.760 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-26 2021-03-02T12:12:24.760 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-26 zones={test-zone2, test-zone1} 2021-03-02T12:12:24.770 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:24.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:24.819 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:24.827 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:24.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:24.892 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-26 zones={test-zone2, test-zone1} 2021-03-02T12:12:24.893 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-26 zones={test-zone3, test-zone1} 2021-03-02T12:12:24.902 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:24.952 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:25.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:25.116 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:25.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:25.180 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-26 zones={test-zone3, test-zone1} 2021-03-02T12:12:25.180 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-26 zones={test-zone3, test-zone2} 2021-03-02T12:12:25.198 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:12:25.252 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:25.253 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:12:25.289 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-02T12:12:25.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-02T12:12:25.382 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-26 zones={test-zone3, test-zone2} 2021-03-02T12:12:25.383 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-03-02T12:12:25.383 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-27 2021-03-02T12:12:25.425 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-28 2021-03-02T12:12:25.501 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-29 2021-03-02T12:12:25.792 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_versioning ... ok 2021-03-02T12:12:25.793 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-30 2021-03-02T12:12:25.827 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-31 2021-03-02T12:12:25.893 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-32 2021-03-02T12:12:26.177 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_acl ... ok 2021-03-02T12:12:26.178 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-33 2021-03-02T12:12:26.215 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-34 2021-03-02T12:12:26.276 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-35 2021-03-02T12:12:28.539 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_cors ... ok 2021-03-02T12:12:28.539 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-36 2021-03-02T12:12:28.576 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-37 2021-03-02T12:12:28.632 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-38 2021-03-02T12:12:28.689 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:28.690 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:28.690 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:28.727 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:28.728 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:28.929 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:12:28.962 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687148.669144_334.1","last_update":"2021-03-02T12:12:28.669144Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:12:28.963 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:12:28.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:28.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:28.965 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:29.006 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:29.007 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:29.218 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687148.476105_321.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:28.476105Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:29.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687148.669144_334.1 target=1_1614687148.476105_321.1 2021-03-02T12:12:34.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:34.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:34.222 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:34.267 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:34.268 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:34.476 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687148.669144_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:28.669144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:34.477 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:12:34.477 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:12:34.478 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:34.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:34.479 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:34.522 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:34.759 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687148.669144_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:28.669144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:34.759 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:12:34.760 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:12:34.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:34.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:34.762 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:34.800 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:34.951 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:35.051 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687148.669144_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:28.669144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:35.109 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:12:35.479 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-03-02T12:12:35.479 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-39 2021-03-02T12:12:35.510 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-40 2021-03-02T12:12:35.565 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-41 2021-03-02T12:12:35.628 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:35.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:35.629 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:35.690 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:35.691 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:35.862 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=34dc26a2-30eb-4d82-a327-548c712f3e72 2021-03-02T12:12:35.897 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614686849.835843_128.1","last_update":"2021-03-02T12:07:29.835843Z"},{"marker":"1_1614687155.603950_347.1","last_update":"2021-03-02T12:12:35.603950Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:12:35.897 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:12:35.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:35.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:35.899 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:35.943 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:35.943 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:36.159 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687148.669144_334.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:28.669144Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:36.159 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687155.603950_347.1 target=1_1614687148.669144_334.1 2021-03-02T12:12:41.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:41.163 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:41.163 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:41.205 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:41.205 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:41.406 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687155.603950_347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:35.603950Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:41.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:12:41.407 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:12:41.407 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:41.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:41.408 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:41.449 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:41.450 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:41.659 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687155.603950_347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:35.603950Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:41.659 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:12:41.660 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:12:41.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:41.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:12:41.661 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:12:41.704 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:41.705 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:41.969 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"34dc26a2-30eb-4d82-a327-548c712f3e72","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687155.603950_347.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:35.603950Z","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:12:42.007 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:12:42.008 DEBUG:tasks.rgw.c1.client.1:waiting for process to exit 2021-03-02T12:12:42.009 INFO:teuthology.orchestra.run:waiting for 300 2021-03-02T12:12:42.010 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:12:42.007+0000 7f1e41cc1700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 15985) UID: 0 2021-03-02T12:12:42.010 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:12:42.007+0000 7f1e593840c0 -1 shutting down 2021-03-02T12:12:46.734 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 is failed for ~0s 2021-03-02T12:12:48.111 INFO:tasks.rgw.c1.client.1:Stopped 2021-03-02T12:12:48.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:12:48.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:12:48.113 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:12:48.154 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:48.155 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:48.184 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:12:48.178+0000 7ff63d9c8080 0 NOTICE: overriding master zone: 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:12:48.186 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:12:48.182+0000 7ff63d9c8080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:12:48.186 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:12:48.182+0000 7ff63d9c8080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:12:48.186 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:12:48.182+0000 7ff63d9c8080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:12:48.187 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:12:48.182+0000 7ff63d9c8080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:12:48.197 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:12:48.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:12:48.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:12:48.199 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:12:48.280 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:12:48.280 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:12:48.545 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"f31ea438-345b-4e8e-ac2d-a40213013c0a","epoch":1,"predecessor_uuid":"34dc26a2-30eb-4d82-a327-548c712f3e72","sync_status":["1_1614686849.835843_128.1","1_1614687155.603950_347.1","",""],"period_map":{"id":"f31ea438-345b-4e8e-ac2d-a40213013c0a","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"78024174-fc84-48b4-be73-8c69857e8755","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"78024174-fc84-48b4-be73-8c69857e8755","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":3} 2021-03-02T12:12:48.546 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-03-02T12:12:52.340 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.1 has been restored 2021-03-02T12:13:48.552 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-42 2021-03-02T12:13:48.682 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-43 2021-03-02T12:13:48.717 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:13:48.717 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:13:48.718 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:13:48.758 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:13:48.758 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:13:48.920 INFO:teuthology.orchestra.run.smithi036.stderr:No --period given, using current period=f31ea438-345b-4e8e-ac2d-a40213013c0a 2021-03-02T12:13:48.953 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614687228.712852_215.1","last_update":"2021-03-02T12:13:48.712852Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:13:48.954 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-03-02T12:13:48.955 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:13:48.955 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:13:48.956 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:13:48.998 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:13:48.998 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:13:49.219 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f31ea438-345b-4e8e-ac2d-a40213013c0a","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-03-02T12:13:49.219 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=1 master=1_1614687228.712852_215.1 target= 2021-03-02T12:13:54.221 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:13:54.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:13:54.223 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:13:54.266 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:13:54.267 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:13:54.483 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"f31ea438-345b-4e8e-ac2d-a40213013c0a","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":80,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"1_1614687228.712852_215.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:13:48.712852Z","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":80,"complete":80}} 2021-03-02T12:13:54.483 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-03-02T12:13:54.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:13:54.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:13:54.485 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:13:54.564 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:13:54.565 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:13:54.592 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:13:54.591+0000 7fe14c428080 0 NOTICE: overriding master zone: 78024174-fc84-48b4-be73-8c69857e8755 2021-03-02T12:13:54.592 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:13:54.591+0000 7fe14c428080 0 failed reading obj info from .rgw.root:zone_info.0680c8ac-6fa2-460e-aca9-ec1ab695cedd: (2) No such file or directory 2021-03-02T12:13:54.593 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:13:54.591+0000 7fe14c428080 0 WARNING: could not read zone params for zone id=0680c8ac-6fa2-460e-aca9-ec1ab695cedd name=test-zone4 2021-03-02T12:13:54.593 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:13:54.591+0000 7fe14c428080 0 failed reading obj info from .rgw.root:zone_info.78024174-fc84-48b4-be73-8c69857e8755: (2) No such file or directory 2021-03-02T12:13:54.594 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:13:54.591+0000 7fe14c428080 0 WARNING: could not read zone params for zone id=78024174-fc84-48b4-be73-8c69857e8755 name=test-zone2 2021-03-02T12:13:54.608 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:13:54.608 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:13:54.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:13:54.609 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:13:54.691 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:13:54.691 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:13:54.986 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":1,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:13:54.987 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-03-02T12:14:54.989 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-44 2021-03-02T12:14:55.075 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-45 2021-03-02T12:14:55.145 INFO:tasks.rgw.c1.client.1:Restarting daemon 2021-03-02T12:14:55.146 DEBUG:teuthology.orchestra.run.smithi012:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c1 -k /etc/ceph/c1.client.1.keyring --log-file /var/log/ceph/rgw.c1.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.1.stdout 2>&1 2021-03-02T12:14:55.158 INFO:tasks.rgw.c1.client.1:Started 2021-03-02T12:14:55.159 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001 2021-03-02T12:14:55.185 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:14:55.186 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:14:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-02T12:14:55.193 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi012.front.sepia.ceph.com port 8001: Connection refused 2021-03-02T12:14:56.194 DEBUG:teuthology.orchestra.run.smithi012:> curl http://smithi012.front.sepia.ceph.com:8001 2021-03-02T12:14:56.217 INFO:teuthology.orchestra.run.smithi012.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-02T12:14:56.217 INFO:teuthology.orchestra.run.smithi012.stderr: Dload Upload Total Spent Left Speed 2021-03-02T12:14:56.223 INFO:teuthology.orchestra.run.smithi012.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-02T12:14:56.223 INFO:teuthology.orchestra.run.smithi012.stdout:anonymous 2021-03-02T12:14:56.223 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:56.224 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:56.225 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:14:56.307 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:14:56.307 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:14:56.488 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:14:56.521 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614687295.125219_226.1","last_update":"2021-03-02T12:14:55.125219Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:14:56.522 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:14:56.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:56.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:56.524 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:14:56.563 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:14:56.563 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:14:56.743 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687295.125219_226.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:14:55.125219Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:14:56.744 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:14:56.744 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:14:56.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:56.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:56.746 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:14:56.787 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:14:56.788 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:14:57.003 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687295.125219_226.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:14:55.125219Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:14:57.004 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:14:57.005 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:14:57.006 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:57.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:14:57.008 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:14:57.050 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:14:57.051 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:14:57.314 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687295.125219_226.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:14:55.125219Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:14:57.314 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:14:57.315 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-39 zones={test-zone2, test-zone1} 2021-03-02T12:14:57.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:57.383 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:57.446 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-39 zones={test-zone2, test-zone1} 2021-03-02T12:14:57.447 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-39 zones={test-zone3, test-zone1} 2021-03-02T12:14:57.462 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:57.510 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:57.577 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-39 zones={test-zone3, test-zone1} 2021-03-02T12:14:57.578 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-39 zones={test-zone1, test-zone2} 2021-03-02T12:14:57.589 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:57.635 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:57.696 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-39 zones={test-zone1, test-zone2} 2021-03-02T12:14:57.697 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-39 zones={test-zone3, test-zone2} 2021-03-02T12:14:57.707 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:57.755 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:57.816 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-39 zones={test-zone3, test-zone2} 2021-03-02T12:14:57.817 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-39 zones={test-zone1, test-zone3} 2021-03-02T12:14:57.830 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:57.879 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:57.943 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-39 zones={test-zone1, test-zone3} 2021-03-02T12:14:57.944 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-39 zones={test-zone2, test-zone3} 2021-03-02T12:14:57.953 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.007 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.078 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-39 zones={test-zone2, test-zone3} 2021-03-02T12:14:58.079 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-40 zones={test-zone2, test-zone1} 2021-03-02T12:14:58.093 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.139 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-40 zones={test-zone2, test-zone1} 2021-03-02T12:14:58.207 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-40 zones={test-zone3, test-zone1} 2021-03-02T12:14:58.217 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.267 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.333 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-40 zones={test-zone3, test-zone1} 2021-03-02T12:14:58.334 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-40 zones={test-zone1, test-zone2} 2021-03-02T12:14:58.345 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.399 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.465 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-40 zones={test-zone1, test-zone2} 2021-03-02T12:14:58.466 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-40 zones={test-zone3, test-zone2} 2021-03-02T12:14:58.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.531 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.594 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-40 zones={test-zone3, test-zone2} 2021-03-02T12:14:58.595 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-40 zones={test-zone1, test-zone3} 2021-03-02T12:14:58.609 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.659 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.722 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-40 zones={test-zone1, test-zone3} 2021-03-02T12:14:58.723 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-40 zones={test-zone2, test-zone3} 2021-03-02T12:14:58.736 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.783 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.848 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-40 zones={test-zone2, test-zone3} 2021-03-02T12:14:58.849 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-41 zones={test-zone2, test-zone1} 2021-03-02T12:14:58.862 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:58.911 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:58.975 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-41 zones={test-zone2, test-zone1} 2021-03-02T12:14:58.976 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-41 zones={test-zone3, test-zone1} 2021-03-02T12:14:58.986 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.035 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:59.104 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-41 zones={test-zone3, test-zone1} 2021-03-02T12:14:59.105 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-41 zones={test-zone1, test-zone2} 2021-03-02T12:14:59.116 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:59.233 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-41 zones={test-zone1, test-zone2} 2021-03-02T12:14:59.234 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-41 zones={test-zone3, test-zone2} 2021-03-02T12:14:59.245 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.295 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:59.362 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-41 zones={test-zone3, test-zone2} 2021-03-02T12:14:59.364 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-41 zones={test-zone1, test-zone3} 2021-03-02T12:14:59.377 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:59.495 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-41 zones={test-zone1, test-zone3} 2021-03-02T12:14:59.496 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-41 zones={test-zone2, test-zone3} 2021-03-02T12:14:59.507 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.555 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:59.656 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-41 zones={test-zone2, test-zone3} 2021-03-02T12:14:59.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-42 zones={test-zone2, test-zone1} 2021-03-02T12:14:59.692 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.767 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:14:59.849 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-42 zones={test-zone2, test-zone1} 2021-03-02T12:14:59.850 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-42 zones={test-zone3, test-zone1} 2021-03-02T12:14:59.869 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:14:59.927 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.003 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-42 zones={test-zone3, test-zone1} 2021-03-02T12:15:00.004 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-42 zones={test-zone1, test-zone2} 2021-03-02T12:15:00.015 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.075 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.142 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-42 zones={test-zone1, test-zone2} 2021-03-02T12:15:00.143 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-42 zones={test-zone3, test-zone2} 2021-03-02T12:15:00.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.207 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.269 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-42 zones={test-zone3, test-zone2} 2021-03-02T12:15:00.270 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-42 zones={test-zone1, test-zone3} 2021-03-02T12:15:00.281 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.335 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.398 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-42 zones={test-zone1, test-zone3} 2021-03-02T12:15:00.399 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-42 zones={test-zone2, test-zone3} 2021-03-02T12:15:00.410 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.459 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.533 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-42 zones={test-zone2, test-zone3} 2021-03-02T12:15:00.534 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-43 zones={test-zone2, test-zone1} 2021-03-02T12:15:00.544 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.595 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.657 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-43 zones={test-zone2, test-zone1} 2021-03-02T12:15:00.658 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-43 zones={test-zone3, test-zone1} 2021-03-02T12:15:00.668 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.715 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.784 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-43 zones={test-zone3, test-zone1} 2021-03-02T12:15:00.785 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-43 zones={test-zone1, test-zone2} 2021-03-02T12:15:00.795 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.843 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:00.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-43 zones={test-zone1, test-zone2} 2021-03-02T12:15:00.906 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-43 zones={test-zone3, test-zone2} 2021-03-02T12:15:00.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:00.970 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.036 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-43 zones={test-zone3, test-zone2} 2021-03-02T12:15:01.037 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-43 zones={test-zone1, test-zone3} 2021-03-02T12:15:01.047 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.099 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.162 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-43 zones={test-zone1, test-zone3} 2021-03-02T12:15:01.163 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-43 zones={test-zone2, test-zone3} 2021-03-02T12:15:01.172 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.219 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.282 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-43 zones={test-zone2, test-zone3} 2021-03-02T12:15:01.283 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-44 zones={test-zone2, test-zone1} 2021-03-02T12:15:01.295 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.343 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.407 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-44 zones={test-zone2, test-zone1} 2021-03-02T12:15:01.407 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-44 zones={test-zone3, test-zone1} 2021-03-02T12:15:01.417 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.466 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.535 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-44 zones={test-zone3, test-zone1} 2021-03-02T12:15:01.535 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-44 zones={test-zone1, test-zone2} 2021-03-02T12:15:01.544 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.590 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.654 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-44 zones={test-zone1, test-zone2} 2021-03-02T12:15:01.655 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-44 zones={test-zone3, test-zone2} 2021-03-02T12:15:01.667 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.723 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-44 zones={test-zone3, test-zone2} 2021-03-02T12:15:01.791 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-44 zones={test-zone1, test-zone3} 2021-03-02T12:15:01.805 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.855 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:01.924 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-44 zones={test-zone1, test-zone3} 2021-03-02T12:15:01.925 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-44 zones={test-zone2, test-zone3} 2021-03-02T12:15:01.935 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:01.983 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:02.051 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-44 zones={test-zone2, test-zone3} 2021-03-02T12:15:02.311 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-45 zones={test-zone2, test-zone1} 2021-03-02T12:15:02.323 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:02.371 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:02.441 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-45 zones={test-zone2, test-zone1} 2021-03-02T12:15:02.442 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-45 zones={test-zone3, test-zone1} 2021-03-02T12:15:02.452 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:02.503 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:02.570 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-45 zones={test-zone3, test-zone1} 2021-03-02T12:15:02.867 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-45 zones={test-zone1, test-zone2} 2021-03-02T12:15:02.880 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:02.935 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:03.012 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-45 zones={test-zone1, test-zone2} 2021-03-02T12:15:03.134 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-45 zones={test-zone3, test-zone2} 2021-03-02T12:15:03.146 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:03.195 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:03.263 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-45 zones={test-zone3, test-zone2} 2021-03-02T12:15:03.393 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-45 zones={test-zone1, test-zone3} 2021-03-02T12:15:03.406 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:03.556 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:03.626 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-45 zones={test-zone1, test-zone3} 2021-03-02T12:15:03.663 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-45 zones={test-zone2, test-zone3} 2021-03-02T12:15:03.676 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:15:03.726 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:15:03.790 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-45 zones={test-zone2, test-zone3} 2021-03-02T12:15:03.940 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:03.941 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:03.942 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:03.983 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:03.984 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:04.236 INFO:teuthology.orchestra.run.smithi012.stdout:[{"id":"1_1614686695.645340_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.645340866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"write"}}},{"id":"1_1614686695.652801_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.652801590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.615713_42.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.615713221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.643227_43.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.643227882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.644566_44.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.644566738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.648062_45.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.648062012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.665746_47.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.665746063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.670212_48.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.670212467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.671473_49.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.671473805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.677429_50.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.677429316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.775929_51.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.775929928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.794814_52.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.794814662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.796274_53.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.796274025Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.800003_54.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.800003193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.625039_56.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.625039446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.632766_57.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.632766532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.637002_58.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.637002063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.645004_59.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.645004492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.671551_61.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.671551107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.677369_62.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.677369559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.679211_63.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.679211537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.684963_64.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.684963976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.720220_65.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.720220693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.725839_66.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.725839978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.727649_67.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.727649176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.731624_68.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.731624093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.885972_70.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.885972749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.892224_71.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.892224477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.894079_72.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.894079707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.899004_73.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.899004960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.925605_75.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.925605859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.930697_76.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.930697519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.932844_77.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.932844207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.936806_78.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.936806115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.965967_79.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.965967230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.971362_80.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.971362029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.973375_81.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.973375179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.977435_82.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.977435108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.004599_83.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.004599564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.009376_84.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.009376327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.011460_85.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.011460742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.014836_86.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.014836963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.064402_88.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.064402665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.071999_89.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.071999501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.073945_90.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.073945362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.077703_91.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.077703838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.180977_92.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.180977436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.185899_93.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.185899455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.188337_94.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.188337240Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.191573_95.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.191573647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.292769_96.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.292769785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.297290_97.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.297290460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.299040_98.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.299040975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.304332_99.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.304332790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.352638_101.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.352638582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.357914_102.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.357914668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.359860_103.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.359860577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.363314_104.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.363314190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.451020_105.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.451020100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.456967_106.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.456967327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.461011_107.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.461011020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.465803_108.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.465803262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686843.051667_111.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.051667747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.057594_112.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.057594671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.059596_113.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.059596706Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.065311_114.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.065311192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.088545_115.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.088545675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.094375_116.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.094375564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.098742_117.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.098742100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.107384_118.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.107384568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.156961_119.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.156961422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.162492_120.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.162492955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.164338_121.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.164338384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.170580_122.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.170580134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686849.694632_123.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.694632053Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.700258_124.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.700258113Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.751793_125.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.751793555Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.756777_126.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.756777741Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.829404_127.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.829404151Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.835843_128.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.835843691Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686856.805763_40.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.805763015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.811355_49.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.811355002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.813114_50.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.813114281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.817466_51.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.817466217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.836751_52.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.836751894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.843032_53.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.843032140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.844959_54.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.844959370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.850380_55.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.850380069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.890380_56.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.890380046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.895843_57.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.895843243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.898010_58.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.898010952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.903336_59.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.903336536Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.414408_60.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.414408868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.420819_67.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.420819225Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.422460_68.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.422460178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.426796_69.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.426796820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.449283_70.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.449283626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.455622_71.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.455622547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.457596_72.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.457596951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.463047_73.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.463047131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.510142_74.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.510142743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.515656_75.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.515656579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.517850_76.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.517850413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.523377_77.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.523377655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.285355_78.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.285355776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.290633_92.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.290633460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.292214_93.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.292214930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.296289_94.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.296289014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.326350_95.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.326350307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.332420_96.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.332420263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.334384_97.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.334384011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.338853_98.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.338853212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.385705_99.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.385705716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.390895_100.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.390895611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.392577_101.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.392577164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.396786_102.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.396786386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.431353_103.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.431353179Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.436811_104.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.436811566Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.457467_105.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.457467852Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.463747_106.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.463747795Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.482414_107.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.482414859Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.489696_108.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.489696090Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686992.231199_109.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.231199541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.236518_148.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.236518428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.238095_149.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.238095067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.242234_150.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.242234074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.250063_151.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.250063730Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"write"}}},{"id":"1_1614686992.254509_152.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.254509677Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687048.927852_153.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.927852881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.933178_170.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.933178434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687048.934819_171.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.934819444Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.939900_172.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.939900707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687066.110028_173.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.110028907Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"write"}}},{"id":"1_1614687066.118540_180.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.118540274Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687082.857508_181.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.857508353Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"write"}}},{"id":"1_1614687082.865578_188.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.865578289Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687100.551668_189.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.551668868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.558584_196.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.558584535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.560497_197.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.560497405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.566574_198.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.566574478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.594151_199.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.594151983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.600221_200.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.600221449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.602328_201.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.602328161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.607380_202.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.607380208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.651896_203.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.651896389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.656847_204.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.656847432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.658880_205.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.658880778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.663478_206.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.663478421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.690169_207.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.690169370Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.697222_208.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.697222480Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.702823_209.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.702823362Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.708928_210.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.708928019Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.719562_211.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.719562340Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.726538_212.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.726538039Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687123.627388_213.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.627388233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.632978_230.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.632978999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.634766_231.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.634766773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.638946_232.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.638946474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.662803_233.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.662803249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.668763_234.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.668763928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.670815_235.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.670815274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.677238_236.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.677238435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.134190_237.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.134190512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.138376_238.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.138376776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.139948_239.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.139948864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.143864_240.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.143864424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.163075_241.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.163075361Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.167198_242.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.167198373Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.170324_243.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.170324827Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.175285_244.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.175285221Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.179752_245.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.179752628Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.184992_246.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.184992294Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.195330_247.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.195330837Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.201285_248.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.201285281Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.213248_249.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.213248419Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.220609_250.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.220609211Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.232654_251.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.232654845Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.239450_252.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.239450289Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687145.406615_253.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.406615069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.413068_270.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.413068923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.414837_271.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.414837191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.421208_272.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.421208744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.463012_273.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.463012675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.469059_274.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.469059669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.471025_275.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.471025731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.479835_276.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.479835665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.526656_277.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.526656107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.531205_278.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.531205812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.532929_279.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.532929269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.537279_280.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.537279575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.560206_281.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.560206005Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.567106_282.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.567106177Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.634296_283.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.634296063Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.643110_284.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.643110732Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.718087_285.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.718087684Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.724175_286.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.724175448Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.814456_287.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.814456955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.819275_288.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.819275823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.820938_289.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.820938445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.824902_290.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.824902097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.858141_291.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.858141427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.863551_292.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.863551291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.865435_293.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.865435154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.870065_294.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.870065720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.914651_295.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.914651330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.920899_296.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.920899941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.923100_297.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.923100886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.929627_298.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.929627740Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.968549_299.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.968549866Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.974256_300.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.974256807Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.037055_301.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.037055809Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.045816_302.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.045816564Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.113278_303.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.113278392Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.121945_304.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.121945552Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.201224_305.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.201224008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.205818_306.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.205818499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.207488_307.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.207488899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.212305_308.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.212305125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.239750_309.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.239750124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.245547_310.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.245547037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.247618_311.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.247618811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.254570_312.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.254570777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.304448_313.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.304448872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.310041_314.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.310041493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.312216_315.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.312216687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.318822_316.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.318822014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.345118_317.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.345118650Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.351133_318.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.351133455Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687147.405927_319.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.405927875Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"write"}}},{"id":"1_1614687147.414253_320.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.414253436Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.476105_321.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.476105601Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"write"}}},{"id":"1_1614687148.485649_322.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.485649134Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.561457_323.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.561457055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.566448_324.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.566448383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.568463_325.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.568463823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.572857_326.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.572857162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.600166_327.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.600166130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.605568_328.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.605568203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.607290_329.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.607290744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.611670_330.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.611670219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.654576_331.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.654576255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.660792_332.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.660792070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.662881_333.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.662881571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.669144_334.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.669144980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.493688_335.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.493688601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.499525_337.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.499525622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.501571_338.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.501571886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.506614_339.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.506614046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.535090_340.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.535090122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.539950_341.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.539950269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.541717_342.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.541717382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.546130_343.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.546130857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.589209_344.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.589209281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.595705_345.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.595705087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.597587_346.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.597587656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.603950_347.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.603950195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:04.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:04.245 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:04.247 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:04.290 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:04.291 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:04.513 INFO:teuthology.orchestra.run.smithi036.stdout:[{"id":"1_1614686695.645340_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.645340866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"write"}}},{"id":"1_1614686695.652801_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.652801590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.615713_42.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.615713221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.643227_43.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.643227882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.644566_44.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.644566738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.648062_45.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.648062012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.665746_47.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.665746063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.670212_48.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.670212467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.671473_49.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.671473805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.677429_50.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.677429316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.775929_51.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.775929928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.794814_52.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.794814662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.796274_53.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.796274025Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.800003_54.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.800003193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.625039_56.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.625039446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.632766_57.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.632766532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.637002_58.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.637002063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.645004_59.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.645004492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.671551_61.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.671551107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.677369_62.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.677369559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.679211_63.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.679211537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.684963_64.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.684963976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.720220_65.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.720220693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.725839_66.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.725839978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.727649_67.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.727649176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.731624_68.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.731624093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.885972_70.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.885972749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.892224_71.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.892224477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.894079_72.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.894079707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.899004_73.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.899004960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.925605_75.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.925605859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.930697_76.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.930697519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.932844_77.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.932844207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.936806_78.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.936806115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.965967_79.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.965967230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.971362_80.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.971362029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.973375_81.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.973375179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.977435_82.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.977435108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.004599_83.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.004599564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.009376_84.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.009376327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.011460_85.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.011460742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.014836_86.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.014836963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.064402_88.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.064402665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.071999_89.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.071999501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.073945_90.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.073945362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.077703_91.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.077703838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.180977_92.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.180977436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.185899_93.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.185899455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.188337_94.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.188337240Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.191573_95.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.191573647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.292769_96.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.292769785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.297290_97.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.297290460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.299040_98.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.299040975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.304332_99.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.304332790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.352638_101.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.352638582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.357914_102.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.357914668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.359860_103.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.359860577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.363314_104.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.363314190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.451020_105.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.451020100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.456967_106.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.456967327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.461011_107.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.461011020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.465803_108.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.465803262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686843.051667_111.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.051667747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.057594_112.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.057594671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.059596_113.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.059596706Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.065311_114.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.065311192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.088545_115.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.088545675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.094375_116.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.094375564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.098742_117.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.098742100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.107384_118.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.107384568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.156961_119.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.156961422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.162492_120.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.162492955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.164338_121.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.164338384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.170580_122.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.170580134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686849.694632_123.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.694632053Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.700258_124.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.700258113Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.751793_125.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.751793555Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.756777_126.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.756777741Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.829404_127.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.829404151Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.835843_128.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.835843691Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686856.805763_40.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.805763015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.811355_49.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.811355002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.813114_50.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.813114281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.817466_51.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.817466217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.836751_52.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.836751894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.843032_53.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.843032140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.844959_54.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.844959370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.850380_55.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.850380069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.890380_56.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.890380046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.895843_57.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.895843243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.898010_58.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.898010952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.903336_59.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.903336536Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.414408_60.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.414408868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.420819_67.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.420819225Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.422460_68.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.422460178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.426796_69.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.426796820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.449283_70.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.449283626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.455622_71.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.455622547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.457596_72.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.457596951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.463047_73.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.463047131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.510142_74.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.510142743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.515656_75.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.515656579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.517850_76.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.517850413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.523377_77.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.523377655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.285355_78.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.285355776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.290633_92.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.290633460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.292214_93.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.292214930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.296289_94.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.296289014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.326350_95.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.326350307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.332420_96.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.332420263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.334384_97.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.334384011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.338853_98.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.338853212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.385705_99.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.385705716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.390895_100.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.390895611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.392577_101.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.392577164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.396786_102.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.396786386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.431353_103.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.431353179Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.436811_104.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.436811566Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.457467_105.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.457467852Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.463747_106.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.463747795Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.482414_107.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.482414859Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.489696_108.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.489696090Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686992.231199_109.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.231199541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.236518_148.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.236518428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.238095_149.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.238095067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.242234_150.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.242234074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.250063_151.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.250063730Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"write"}}},{"id":"1_1614686992.254509_152.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.254509677Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687048.927852_153.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.927852881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.933178_170.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.933178434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687048.934819_171.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.934819444Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.939900_172.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.939900707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687066.110028_173.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.110028907Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"write"}}},{"id":"1_1614687066.118540_180.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.118540274Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687082.857508_181.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.857508353Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"write"}}},{"id":"1_1614687082.865578_188.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.865578289Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687100.551668_189.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.551668868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.558584_196.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.558584535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.560497_197.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.560497405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.566574_198.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.566574478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.594151_199.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.594151983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.600221_200.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.600221449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.602328_201.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.602328161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.607380_202.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.607380208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.651896_203.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.651896389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.656847_204.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.656847432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.658880_205.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.658880778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.663478_206.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.663478421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.690169_207.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.690169370Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.697222_208.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.697222480Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.702823_209.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.702823362Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.708928_210.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.708928019Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.719562_211.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.719562340Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.726538_212.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.726538039Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687123.627388_213.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.627388233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.632978_230.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.632978999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.634766_231.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.634766773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.638946_232.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.638946474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.662803_233.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.662803249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.668763_234.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.668763928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.670815_235.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.670815274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.677238_236.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.677238435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.134190_237.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.134190512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.138376_238.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.138376776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.139948_239.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.139948864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.143864_240.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.143864424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.163075_241.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.163075361Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.167198_242.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.167198373Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.170324_243.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.170324827Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.175285_244.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.175285221Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.179752_245.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.179752628Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.184992_246.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.184992294Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.195330_247.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.195330837Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.201285_248.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.201285281Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.213248_249.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.213248419Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.220609_250.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.220609211Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.232654_251.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.232654845Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.239450_252.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.239450289Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687145.406615_253.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.406615069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.413068_270.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.413068923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.414837_271.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.414837191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.421208_272.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.421208744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.463012_273.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.463012675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.469059_274.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.469059669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.471025_275.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.471025731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.479835_276.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.479835665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.526656_277.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.526656107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.531205_278.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.531205812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.532929_279.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.532929269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.537279_280.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.537279575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.560206_281.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.560206005Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.567106_282.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.567106177Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.634296_283.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.634296063Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.643110_284.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.643110732Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.718087_285.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.718087684Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.724175_286.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.724175448Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.814456_287.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.814456955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.819275_288.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.819275823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.820938_289.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.820938445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.824902_290.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.824902097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.858141_291.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.858141427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.863551_292.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.863551291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.865435_293.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.865435154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.870065_294.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.870065720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.914651_295.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.914651330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.920899_296.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.920899941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.923100_297.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.923100886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.929627_298.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.929627740Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.968549_299.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.968549866Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.974256_300.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.974256807Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.037055_301.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.037055809Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.045816_302.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.045816564Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.113278_303.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.113278392Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.121945_304.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.121945552Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.201224_305.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.201224008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.205818_306.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.205818499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.207488_307.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.207488899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.212305_308.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.212305125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.239750_309.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.239750124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.245547_310.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.245547037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.247618_311.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.247618811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.254570_312.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.254570777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.304448_313.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.304448872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.310041_314.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.310041493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.312216_315.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.312216687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.318822_316.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.318822014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.345118_317.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.345118650Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.351133_318.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.351133455Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687147.405927_319.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.405927875Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"write"}}},{"id":"1_1614687147.414253_320.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.414253436Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.476105_321.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.476105601Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"write"}}},{"id":"1_1614687148.485649_322.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.485649134Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.561457_323.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.561457055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.566448_324.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.566448383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.568463_325.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.568463823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.572857_326.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.572857162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.600166_327.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.600166130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.605568_328.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.605568203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.607290_329.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.607290744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.611670_330.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.611670219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.654576_331.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.654576255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.660792_332.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.660792070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.662881_333.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.662881571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.669144_334.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.669144980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.493688_335.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.493688601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.499525_337.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.499525622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.501571_338.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.501571886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.506614_339.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.506614046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.535090_340.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.535090122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.539950_341.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.539950269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.541717_342.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.541717382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.546130_343.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.546130857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.589209_344.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.589209281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.595705_345.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.595705087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.597587_346.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.597587656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.603950_347.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.603950195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:04.653 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:04.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:04.654 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:04.696 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:04.696 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:04.942 INFO:teuthology.orchestra.run.smithi012.stdout:[{"id":"1_1614686695.645340_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.645340866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"write"}}},{"id":"1_1614686695.652801_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.652801590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.615713_42.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.615713221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.643227_43.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.643227882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.644566_44.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.644566738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.648062_45.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.648062012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.665746_47.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.665746063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.670212_48.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.670212467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.671473_49.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.671473805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.677429_50.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.677429316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.775929_51.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.775929928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.794814_52.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.794814662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.796274_53.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.796274025Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.800003_54.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.800003193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.625039_56.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.625039446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.632766_57.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.632766532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.637002_58.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.637002063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.645004_59.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.645004492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.671551_61.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.671551107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.677369_62.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.677369559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.679211_63.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.679211537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.684963_64.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.684963976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.720220_65.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.720220693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.725839_66.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.725839978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.727649_67.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.727649176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.731624_68.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.731624093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.885972_70.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.885972749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.892224_71.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.892224477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.894079_72.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.894079707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.899004_73.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.899004960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.925605_75.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.925605859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.930697_76.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.930697519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.932844_77.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.932844207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.936806_78.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.936806115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.965967_79.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.965967230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.971362_80.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.971362029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.973375_81.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.973375179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.977435_82.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.977435108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.004599_83.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.004599564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.009376_84.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.009376327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.011460_85.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.011460742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.014836_86.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.014836963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.064402_88.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.064402665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.071999_89.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.071999501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.073945_90.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.073945362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.077703_91.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.077703838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.180977_92.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.180977436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.185899_93.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.185899455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.188337_94.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.188337240Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.191573_95.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.191573647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.292769_96.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.292769785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.297290_97.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.297290460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.299040_98.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.299040975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.304332_99.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.304332790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.352638_101.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.352638582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.357914_102.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.357914668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.359860_103.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.359860577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.363314_104.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.363314190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.451020_105.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.451020100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.456967_106.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.456967327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.461011_107.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.461011020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.465803_108.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.465803262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686843.051667_111.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.051667747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.057594_112.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.057594671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.059596_113.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.059596706Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.065311_114.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.065311192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.088545_115.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.088545675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.094375_116.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.094375564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.098742_117.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.098742100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.107384_118.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.107384568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.156961_119.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.156961422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.162492_120.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.162492955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.164338_121.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.164338384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.170580_122.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.170580134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686849.694632_123.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.694632053Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.700258_124.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.700258113Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.751793_125.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.751793555Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.756777_126.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.756777741Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.829404_127.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.829404151Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.835843_128.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.835843691Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686856.805763_40.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.805763015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.811355_49.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.811355002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.813114_50.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.813114281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.817466_51.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.817466217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.836751_52.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.836751894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.843032_53.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.843032140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.844959_54.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.844959370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.850380_55.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.850380069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.890380_56.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.890380046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.895843_57.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.895843243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.898010_58.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.898010952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.903336_59.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.903336536Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.414408_60.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.414408868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.420819_67.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.420819225Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.422460_68.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.422460178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.426796_69.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.426796820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.449283_70.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.449283626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.455622_71.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.455622547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.457596_72.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.457596951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.463047_73.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.463047131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.510142_74.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.510142743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.515656_75.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.515656579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.517850_76.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.517850413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.523377_77.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.523377655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.285355_78.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.285355776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.290633_92.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.290633460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.292214_93.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.292214930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.296289_94.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.296289014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.326350_95.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.326350307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.332420_96.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.332420263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.334384_97.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.334384011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.338853_98.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.338853212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.385705_99.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.385705716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.390895_100.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.390895611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.392577_101.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.392577164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.396786_102.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.396786386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.431353_103.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.431353179Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.436811_104.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.436811566Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.457467_105.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.457467852Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.463747_106.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.463747795Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.482414_107.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.482414859Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.489696_108.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.489696090Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686992.231199_109.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.231199541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.236518_148.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.236518428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.238095_149.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.238095067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.242234_150.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.242234074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.250063_151.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.250063730Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"write"}}},{"id":"1_1614686992.254509_152.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.254509677Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687048.927852_153.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.927852881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.933178_170.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.933178434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687048.934819_171.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.934819444Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.939900_172.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.939900707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687066.110028_173.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.110028907Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"write"}}},{"id":"1_1614687066.118540_180.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.118540274Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687082.857508_181.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.857508353Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"write"}}},{"id":"1_1614687082.865578_188.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.865578289Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687100.551668_189.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.551668868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.558584_196.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.558584535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.560497_197.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.560497405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.566574_198.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.566574478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.594151_199.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.594151983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.600221_200.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.600221449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.602328_201.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.602328161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.607380_202.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.607380208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.651896_203.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.651896389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.656847_204.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.656847432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.658880_205.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.658880778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.663478_206.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.663478421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.690169_207.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.690169370Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.697222_208.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.697222480Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.702823_209.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.702823362Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.708928_210.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.708928019Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.719562_211.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.719562340Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.726538_212.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.726538039Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687123.627388_213.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.627388233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.632978_230.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.632978999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.634766_231.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.634766773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.638946_232.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.638946474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.662803_233.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.662803249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.668763_234.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.668763928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.670815_235.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.670815274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.677238_236.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.677238435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.134190_237.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.134190512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.138376_238.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.138376776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.139948_239.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.139948864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.143864_240.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.143864424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.163075_241.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.163075361Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.167198_242.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.167198373Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.170324_243.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.170324827Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.175285_244.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.175285221Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.179752_245.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.179752628Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.184992_246.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.184992294Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.195330_247.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.195330837Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.201285_248.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.201285281Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.213248_249.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.213248419Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.220609_250.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.220609211Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.232654_251.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.232654845Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.239450_252.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.239450289Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687145.406615_253.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.406615069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.413068_270.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.413068923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.414837_271.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.414837191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.421208_272.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.421208744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.463012_273.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.463012675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.469059_274.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.469059669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.471025_275.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.471025731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.479835_276.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.479835665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.526656_277.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.526656107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.531205_278.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.531205812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.532929_279.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.532929269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.537279_280.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.537279575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.560206_281.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.560206005Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.567106_282.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.567106177Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.634296_283.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.634296063Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.643110_284.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.643110732Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.718087_285.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.718087684Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.724175_286.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.724175448Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.814456_287.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.814456955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.819275_288.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.819275823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.820938_289.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.820938445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.824902_290.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.824902097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.858141_291.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.858141427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.863551_292.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.863551291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.865435_293.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.865435154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.870065_294.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.870065720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.914651_295.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.914651330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.920899_296.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.920899941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.923100_297.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.923100886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.929627_298.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.929627740Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.968549_299.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.968549866Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.974256_300.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.974256807Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.037055_301.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.037055809Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.045816_302.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.045816564Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.113278_303.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.113278392Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.121945_304.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.121945552Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.201224_305.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.201224008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.205818_306.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.205818499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.207488_307.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.207488899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.212305_308.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.212305125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.239750_309.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.239750124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.245547_310.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.245547037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.247618_311.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.247618811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.254570_312.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.254570777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.304448_313.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.304448872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.310041_314.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.310041493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.312216_315.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.312216687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.318822_316.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.318822014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.345118_317.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.345118650Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.351133_318.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.351133455Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687147.405927_319.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.405927875Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"write"}}},{"id":"1_1614687147.414253_320.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.414253436Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.476105_321.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.476105601Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"write"}}},{"id":"1_1614687148.485649_322.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.485649134Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.561457_323.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.561457055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.566448_324.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.566448383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.568463_325.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.568463823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.572857_326.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.572857162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.600166_327.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.600166130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.605568_328.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.605568203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.607290_329.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.607290744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.611670_330.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.611670219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.654576_331.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.654576255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.660792_332.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.660792070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.662881_333.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.662881571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.669144_334.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.669144980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.493688_335.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.493688601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.499525_337.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.499525622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.501571_338.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.501571886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.506614_339.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.506614046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.535090_340.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.535090122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.539950_341.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.539950269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.541717_342.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.541717382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.546130_343.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.546130857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.589209_344.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.589209281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.595705_345.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.595705087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.597587_346.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.597587656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.603950_347.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.603950195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:04.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:04.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:04.950 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:05.006 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:05.007 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:05.215 INFO:teuthology.orchestra.run.smithi036.stdout:[{"id":"1_1614686695.645340_40.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.645340866Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"write"}}},{"id":"1_1614686695.652801_41.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-02T12:04:55.652801590Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9hEMsdsfxcHGnNsvG3Mi9oG","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.615713_42.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.615713221Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.643227_43.1","section":"bucket.instance","name":"majrnq-1:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1","timestamp":"2021-03-02T12:06:57.643227882Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uOGvIUNObt1Ms10DZIXcpWm","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.644566_44.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.644566738Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.648062_45.1","section":"bucket","name":"majrnq-1","timestamp":"2021-03-02T12:06:57.648062012Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZKr1KMB6dSNCOq08btzc3YL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.665746_47.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.665746063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.670212_48.1","section":"bucket.instance","name":"majrnq-2:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2","timestamp":"2021-03-02T12:06:57.670212467Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LDnBM28Ddx6rAazjLDYIQUV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686817.671473_49.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.671473805Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"write"}}},{"id":"1_1614686817.677429_50.1","section":"bucket","name":"majrnq-2","timestamp":"2021-03-02T12:06:57.677429316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_7dwKYbxplY_UIpXrn1GcgHP","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.775929_51.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.775929928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.794814_52.1","section":"bucket.instance","name":"majrnq-3:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3","timestamp":"2021-03-02T12:07:01.794814662Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vLYMs7MqsYo0y_ZpQ2Gphd","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686821.796274_53.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.796274025Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686821.800003_54.1","section":"bucket","name":"majrnq-3","timestamp":"2021-03-02T12:07:01.800003193Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DILKg88mQbrqWiWC5B7xMfR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.625039_56.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.625039446Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.632766_57.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4","timestamp":"2021-03-02T12:07:08.632766532Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wEXet7EyW2DlRhhhrTMm599","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.637002_58.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.637002063Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.645004_59.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:08.645004492Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WeUA9JmYey6Bi71gLJCupAc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.671551_61.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.671551107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.677369_62.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5","timestamp":"2021-03-02T12:07:08.677369559Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UqIiE9Cs5g2Yg773NrmneZ0","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.679211_63.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.679211537Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.684963_64.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:08.684963976Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hTQMpKKpuP1oxLjWS2Vo3c6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.720220_65.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.720220693Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.725839_66.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6","timestamp":"2021-03-02T12:07:08.725839978Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_f3wlgM8eqiEM5DPLTj8TZ0h","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686828.727649_67.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.727649176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"write"}}},{"id":"1_1614686828.731624_68.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:08.731624093Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTTzo0nMSBDORd5vzO-EEO9","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.885972_70.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.885972749Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.892224_71.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.7","timestamp":"2021-03-02T12:07:15.892224477Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_E_JJj9z-7mUM56Z4uQpsHHa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.894079_72.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.894079707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.899004_73.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:15.899004960Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LgSnQfG5kXE9szEslfG7HgW","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.925605_75.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.925605859Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.930697_76.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.8","timestamp":"2021-03-02T12:07:15.930697519Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rrrsenyBJ8FYNoP0UlZdMF7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.932844_77.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.932844207Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.936806_78.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:15.936806115Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oeh3937RR7kfhuu6TOPybpj","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686835.965967_79.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.965967230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.971362_80.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.9","timestamp":"2021-03-02T12:07:15.971362029Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_q3nHN9cHoPYhyvx6OJnDtvI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686835.973375_81.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.973375179Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686835.977435_82.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:15.977435108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eFYHucEBn4CuzKoaOw0MBBQ","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.004599_83.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.004599564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.009376_84.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.10","timestamp":"2021-03-02T12:07:16.009376327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CK5FjIQ3_8P4ju4mITEire8","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.011460_85.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.011460742Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.014836_86.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.014836963Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PTkv42Q3ARNh1Lea59h2Zfc","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.064402_88.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.064402665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.071999_89.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.11","timestamp":"2021-03-02T12:07:16.071999501Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_M7lbIz2Ts8m15GZDb3ptFMR","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.073945_90.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.073945362Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.077703_91.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.077703838Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5TNRTFghxFEXR_Tx8_Jdvnz","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.180977_92.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.180977436Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.185899_93.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.12","timestamp":"2021-03-02T12:07:16.185899455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8C3VSIdpMcrV70VSwZgI31-","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.188337_94.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.188337240Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.191573_95.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.191573647Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_T0Dc3wv7sgtCdR8coDU8wN_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.292769_96.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.292769785Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.297290_97.1","section":"bucket.instance","name":"majrnq-4:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.13","timestamp":"2021-03-02T12:07:16.297290460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UXiZr0HqeKo06Vf2B445MXQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.299040_98.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.299040975Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.304332_99.1","section":"bucket","name":"majrnq-4","timestamp":"2021-03-02T12:07:16.304332790Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_x-GvNxxtvQzf1KY20U2Gpd_","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.352638_101.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.352638582Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.357914_102.1","section":"bucket.instance","name":"majrnq-5:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.14","timestamp":"2021-03-02T12:07:16.357914668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lHTR2Fewg1sE1unhOPobOv6","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.359860_103.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.359860577Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.363314_104.1","section":"bucket","name":"majrnq-5","timestamp":"2021-03-02T12:07:16.363314190Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OYONo2vAnobws1zRuW2BhRL","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686836.451020_105.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.451020100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.456967_106.1","section":"bucket.instance","name":"majrnq-6:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.15","timestamp":"2021-03-02T12:07:16.456967327Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XNfeP3GAuG0NsLxxHzmJJAD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686836.461011_107.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.461011020Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"write"}}},{"id":"1_1614686836.465803_108.1","section":"bucket","name":"majrnq-6","timestamp":"2021-03-02T12:07:16.465803262Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jJQIr0w96u0e1xA3oGSlusY","ver":1},"status":{"status":"abort"}}},{"id":"1_1614686843.051667_111.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.051667747Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.057594_112.1","section":"bucket.instance","name":"majrnq-7:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16","timestamp":"2021-03-02T12:07:23.057594671Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9vqcZuteLAwwBAof3PQjP7y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.059596_113.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.059596706Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.065311_114.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:23.065311192Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.088545_115.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.088545675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.094375_116.1","section":"bucket.instance","name":"majrnq-8:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17","timestamp":"2021-03-02T12:07:23.094375564Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ItaE1d4FOh4TzsSMyAC0fPN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.098742_117.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.098742100Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.107384_118.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:23.107384568Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.156961_119.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.156961422Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.162492_120.1","section":"bucket.instance","name":"majrnq-9:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18","timestamp":"2021-03-02T12:07:23.162492955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Vypw55afz8CnHCsY5a03-Al","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686843.164338_121.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.164338384Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"write"}}},{"id":"1_1614686843.170580_122.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:23.170580134Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686849.694632_123.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.694632053Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.700258_124.1","section":"bucket","name":"majrnq-7","timestamp":"2021-03-02T12:07:29.700258113Z","data":{"read_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":1},"write_version":{"tag":"_jD1eXro_7A9srL8I5-SnJM7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.751793_125.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.751793555Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.756777_126.1","section":"bucket","name":"majrnq-8","timestamp":"2021-03-02T12:07:29.756777741Z","data":{"read_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":1},"write_version":{"tag":"_KW0Jz76LWA7IPRH-wTd5ydU","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686849.829404_127.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.829404151Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"remove"}}},{"id":"1_1614686849.835843_128.1","section":"bucket","name":"majrnq-9","timestamp":"2021-03-02T12:07:29.835843691Z","data":{"read_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":1},"write_version":{"tag":"_0UAq6K5oTuw45NteygSEuyK","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686856.805763_40.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.805763015Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.811355_49.1","section":"bucket.instance","name":"majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19","timestamp":"2021-03-02T12:07:36.811355002Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dPSbqlDBEs7VheiHxKWRf6E","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.813114_50.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.813114281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.817466_51.1","section":"bucket","name":"majrnq-10","timestamp":"2021-03-02T12:07:36.817466217Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_K2scnWynBZAKH85QiVQfGY5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.836751_52.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.836751894Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.843032_53.1","section":"bucket.instance","name":"majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20","timestamp":"2021-03-02T12:07:36.843032140Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_uLryIwRPR2dTie1VnCcWJx7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.844959_54.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.844959370Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.850380_55.1","section":"bucket","name":"majrnq-11","timestamp":"2021-03-02T12:07:36.850380069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CEmuTMgc7wuWJKWZFcP8Oav","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.890380_56.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.890380046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.895843_57.1","section":"bucket.instance","name":"majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21","timestamp":"2021-03-02T12:07:36.895843243Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ycxuPhahcgluafElIJ-3E1w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686856.898010_58.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.898010952Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"write"}}},{"id":"1_1614686856.903336_59.1","section":"bucket","name":"majrnq-12","timestamp":"2021-03-02T12:07:36.903336536Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GMswv6NSL1D1N1y_5jbu2Do","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.414408_60.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.414408868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.420819_67.1","section":"bucket.instance","name":"majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22","timestamp":"2021-03-02T12:07:50.420819225Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WmTwVuKhi17U82vS3KrwNrU","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.422460_68.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.422460178Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.426796_69.1","section":"bucket","name":"majrnq-13","timestamp":"2021-03-02T12:07:50.426796820Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lZfbU-MfDUu5KPx18GcdjyI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.449283_70.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.449283626Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.455622_71.1","section":"bucket.instance","name":"majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23","timestamp":"2021-03-02T12:07:50.455622547Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QHpU3W3lNi6RuBQ51Xv7T47","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.457596_72.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.457596951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.463047_73.1","section":"bucket","name":"majrnq-14","timestamp":"2021-03-02T12:07:50.463047131Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9rjfInpfBtBIKvICv5ctmy1","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.510142_74.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.510142743Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.515656_75.1","section":"bucket.instance","name":"majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24","timestamp":"2021-03-02T12:07:50.515656579Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Go8D4Bolj9sQEBprQRQpVAl","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686870.517850_76.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.517850413Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686870.523377_77.1","section":"bucket","name":"majrnq-15","timestamp":"2021-03-02T12:07:50.523377655Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Xvv_yQ48Bo0yn83F4cf_IMv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.285355_78.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.285355776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.290633_92.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.290633460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.292214_93.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.292214930Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.296289_94.1","section":"bucket","name":"majrnq-16","timestamp":"2021-03-02T12:08:26.296289014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_OOdnOPQvUb82UdIwJp9T3jL","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.326350_95.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.326350307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.332420_96.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.332420263Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.334384_97.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.334384011Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.338853_98.1","section":"bucket","name":"majrnq-17","timestamp":"2021-03-02T12:08:26.338853212Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_itYg8zikkGNyDavcjEzY-Rz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.385705_99.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.385705716Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.390895_100.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.390895611Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.392577_101.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.392577164Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"write"}}},{"id":"1_1614686906.396786_102.1","section":"bucket","name":"majrnq-18","timestamp":"2021-03-02T12:08:26.396786386Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_dAe7FOJth_vnKW89KYok8Jv","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686906.431353_103.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.431353179Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.436811_104.1","section":"bucket.instance","name":"majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25","timestamp":"2021-03-02T12:08:26.436811566Z","data":{"read_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":1},"write_version":{"tag":"_p4irSN-USQvmegnPgw36LCb","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.457467_105.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.457467852Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.463747_106.1","section":"bucket.instance","name":"majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26","timestamp":"2021-03-02T12:08:26.463747795Z","data":{"read_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":1},"write_version":{"tag":"_6vUh6jK8rQ_KlIOf-lDSGz7","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686906.482414_107.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.482414859Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"write"}}},{"id":"1_1614686906.489696_108.1","section":"bucket.instance","name":"majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27","timestamp":"2021-03-02T12:08:26.489696090Z","data":{"read_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":1},"write_version":{"tag":"_y-Neo0WiRUzrOlUsDsR1QwE","ver":2},"status":{"status":"complete"}}},{"id":"1_1614686992.231199_109.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.231199541Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.236518_148.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.236518428Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.238095_149.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.238095067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"write"}}},{"id":"1_1614686992.242234_150.1","section":"bucket","name":"majrnq-19","timestamp":"2021-03-02T12:09:52.242234074Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CLIhRaqACdp7e3ZJFkSKNdr","ver":1},"status":{"status":"complete"}}},{"id":"1_1614686992.250063_151.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.250063730Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"write"}}},{"id":"1_1614686992.254509_152.1","section":"bucket.instance","name":"majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28","timestamp":"2021-03-02T12:09:52.254509677Z","data":{"read_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":1},"write_version":{"tag":"_zrep23ClzAYz66MZNn6CDCS","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687048.927852_153.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.927852881Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.933178_170.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:10:48.933178434Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687048.934819_171.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.934819444Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687048.939900_172.1","section":"bucket","name":"majrnq-20","timestamp":"2021-03-02T12:10:48.939900707Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_95a-gvsIfmbYBjnZ7oc3BMn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687066.110028_173.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.110028907Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"write"}}},{"id":"1_1614687066.118540_180.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:06.118540274Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":1},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687082.857508_181.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.857508353Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"write"}}},{"id":"1_1614687082.865578_188.1","section":"bucket.instance","name":"majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29","timestamp":"2021-03-02T12:11:22.865578289Z","data":{"read_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":2},"write_version":{"tag":"_I6VXli3xH5WRcC42TktCYtV","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687100.551668_189.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.551668868Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.558584_196.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.558584535Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.560497_197.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.560497405Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.566574_198.1","section":"bucket","name":"majrnq-21","timestamp":"2021-03-02T12:11:40.566574478Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_u604_wrzhq0IfCQheusNQNn","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.594151_199.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.594151983Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.600221_200.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.600221449Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.602328_201.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.602328161Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.607380_202.1","section":"bucket","name":"majrnq-22","timestamp":"2021-03-02T12:11:40.607380208Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_GpIL06FD6WTbxOhqYaPX7yD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.651896_203.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.651896389Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.656847_204.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.656847432Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.658880_205.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.658880778Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"write"}}},{"id":"1_1614687100.663478_206.1","section":"bucket","name":"majrnq-23","timestamp":"2021-03-02T12:11:40.663478421Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_my3PaZY9JcrIrwvl_goxD8O","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687100.690169_207.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.690169370Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.697222_208.1","section":"bucket.instance","name":"majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30","timestamp":"2021-03-02T12:11:40.697222480Z","data":{"read_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":1},"write_version":{"tag":"_Dv6lsXNshrdyam0nZY8K1jj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.702823_209.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.702823362Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.708928_210.1","section":"bucket.instance","name":"majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31","timestamp":"2021-03-02T12:11:40.708928019Z","data":{"read_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":1},"write_version":{"tag":"_UUiqMsYw33K9kzHwvpV0aGM","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687100.719562_211.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.719562340Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"write"}}},{"id":"1_1614687100.726538_212.1","section":"bucket.instance","name":"majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32","timestamp":"2021-03-02T12:11:40.726538039Z","data":{"read_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":1},"write_version":{"tag":"_HBED2bFLoUUQ4SYt62G5Qv5","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687123.627388_213.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.627388233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.632978_230.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:03.632978999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.634766_231.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.634766773Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.638946_232.1","section":"bucket","name":"majrnq-24","timestamp":"2021-03-02T12:12:03.638946474Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oaOs-yuTGDNF_5fxA9uairV","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.662803_233.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.662803249Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.668763_234.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:03.668763928Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687123.670815_235.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.670815274Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"write"}}},{"id":"1_1614687123.677238_236.1","section":"bucket","name":"majrnq-25","timestamp":"2021-03-02T12:12:03.677238435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__Nocxqohow_qxj4IXMBzbbY","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.134190_237.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.134190512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.138376_238.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.138376776Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.139948_239.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.139948864Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"write"}}},{"id":"1_1614687124.143864_240.1","section":"bucket","name":"majrnq-26","timestamp":"2021-03-02T12:12:04.143864424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__ay6h0CFy64gLHuuHtNgz4K","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687124.163075_241.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.163075361Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.167198_242.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.167198373Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":1},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.170324_243.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.170324827Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.175285_244.1","section":"bucket.instance","name":"majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33","timestamp":"2021-03-02T12:12:04.175285221Z","data":{"read_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":2},"write_version":{"tag":"_eRJaXfKxwZ7kbHzcHmVy7Ud","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.179752_245.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.179752628Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.184992_246.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.184992294Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":1},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.195330_247.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.195330837Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.201285_248.1","section":"bucket.instance","name":"majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34","timestamp":"2021-03-02T12:12:04.201285281Z","data":{"read_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":2},"write_version":{"tag":"_0UPxaL5nLWVmSQdjQmvlUXY","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687124.213248_249.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.213248419Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"write"}}},{"id":"1_1614687124.220609_250.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.220609211Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":1},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687124.232654_251.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.232654845Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"write"}}},{"id":"1_1614687124.239450_252.1","section":"bucket.instance","name":"majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35","timestamp":"2021-03-02T12:12:04.239450289Z","data":{"read_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":2},"write_version":{"tag":"_BrdvmzHYpHF66EDhlm7Qxbx","ver":3},"status":{"status":"complete"}}},{"id":"1_1614687145.406615_253.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.406615069Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.413068_270.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.413068923Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.414837_271.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.414837191Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.421208_272.1","section":"bucket","name":"majrnq-27","timestamp":"2021-03-02T12:12:25.421208744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ppxkkXY3cprlkACBR1i9X4H","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.463012_273.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.463012675Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.469059_274.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.469059669Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.471025_275.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.471025731Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.479835_276.1","section":"bucket","name":"majrnq-28","timestamp":"2021-03-02T12:12:25.479835665Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2OMrsPwrtwgsLykR53iewJz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.526656_277.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.526656107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.531205_278.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.531205812Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.532929_279.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.532929269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.537279_280.1","section":"bucket","name":"majrnq-29","timestamp":"2021-03-02T12:12:25.537279575Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NZaX4bmANKAKEiZbknqdtMc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.560206_281.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.560206005Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.567106_282.1","section":"bucket.instance","name":"majrnq-27:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36","timestamp":"2021-03-02T12:12:25.567106177Z","data":{"read_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":1},"write_version":{"tag":"_1mWw77xlnhqk0b4Wr6DZLbu","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.634296_283.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.634296063Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.643110_284.1","section":"bucket.instance","name":"majrnq-28:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37","timestamp":"2021-03-02T12:12:25.643110732Z","data":{"read_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":1},"write_version":{"tag":"_CcqQk7K2jBYPaxflwgyWcVW","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.718087_285.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.718087684Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.724175_286.1","section":"bucket.instance","name":"majrnq-29:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38","timestamp":"2021-03-02T12:12:25.724175448Z","data":{"read_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":1},"write_version":{"tag":"_TM6ghnvx0arF_mHC3VR3asa","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687145.814456_287.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.814456955Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.819275_288.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.819275823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.820938_289.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.820938445Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.824902_290.1","section":"bucket","name":"majrnq-30","timestamp":"2021-03-02T12:12:25.824902097Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WBKyYsZPJvC5QAHjvD5l69V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.858141_291.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.858141427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.863551_292.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:25.863551291Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.865435_293.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.865435154Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.870065_294.1","section":"bucket","name":"majrnq-31","timestamp":"2021-03-02T12:12:25.870065720Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Fy_85QGsZUtWnAItOiZlmSX","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.914651_295.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.914651330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.920899_296.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:25.920899941Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.923100_297.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.923100886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687145.929627_298.1","section":"bucket","name":"majrnq-32","timestamp":"2021-03-02T12:12:25.929627740Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CA9GSu0aLLJAqBwGGTXWNb3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687145.968549_299.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.968549866Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"write"}}},{"id":"1_1614687145.974256_300.1","section":"bucket.instance","name":"majrnq-30:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39","timestamp":"2021-03-02T12:12:25.974256807Z","data":{"read_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":1},"write_version":{"tag":"_NHji8VW4W0QfNWPPsfzDVey","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.037055_301.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.037055809Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.045816_302.1","section":"bucket.instance","name":"majrnq-31:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40","timestamp":"2021-03-02T12:12:26.045816564Z","data":{"read_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":1},"write_version":{"tag":"_Lk3dfn7J9R4DpgwN6ca95qz","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.113278_303.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.113278392Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.121945_304.1","section":"bucket.instance","name":"majrnq-32:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41","timestamp":"2021-03-02T12:12:26.121945552Z","data":{"read_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":1},"write_version":{"tag":"_QUjJzqpBlUy5mkTVKWXrMhj","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687146.201224_305.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.201224008Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.205818_306.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.205818499Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.207488_307.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.207488899Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.212305_308.1","section":"bucket","name":"majrnq-33","timestamp":"2021-03-02T12:12:26.212305125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KFXiGJwtraREolF2sTmHJah","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.239750_309.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.239750124Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.245547_310.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:26.245547037Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.247618_311.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.247618811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.254570_312.1","section":"bucket","name":"majrnq-34","timestamp":"2021-03-02T12:12:26.254570777Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_22AkiPGc9O6obDc8rxv7q6y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.304448_313.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.304448872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.310041_314.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:26.310041493Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.312216_315.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.312216687Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"write"}}},{"id":"1_1614687146.318822_316.1","section":"bucket","name":"majrnq-35","timestamp":"2021-03-02T12:12:26.318822014Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_so4c3wAy3DFIPLdWlSm1ZGt","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687146.345118_317.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.345118650Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"write"}}},{"id":"1_1614687146.351133_318.1","section":"bucket.instance","name":"majrnq-33:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42","timestamp":"2021-03-02T12:12:26.351133455Z","data":{"read_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":1},"write_version":{"tag":"_xtPLiUnxMxjbyzTrd-CnXma","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687147.405927_319.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.405927875Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"write"}}},{"id":"1_1614687147.414253_320.1","section":"bucket.instance","name":"majrnq-34:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43","timestamp":"2021-03-02T12:12:27.414253436Z","data":{"read_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":1},"write_version":{"tag":"_XdlSVS-8C-lLD3LPg7EdEZA","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.476105_321.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.476105601Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"write"}}},{"id":"1_1614687148.485649_322.1","section":"bucket.instance","name":"majrnq-35:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44","timestamp":"2021-03-02T12:12:28.485649134Z","data":{"read_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":1},"write_version":{"tag":"_NIXn6Oou4N6HAdU9eP6-ltc","ver":2},"status":{"status":"complete"}}},{"id":"1_1614687148.561457_323.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.561457055Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.566448_324.1","section":"bucket.instance","name":"majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45","timestamp":"2021-03-02T12:12:28.566448383Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_koNIau-ht3XZkulyk_2yzxe","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.568463_325.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.568463823Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.572857_326.1","section":"bucket","name":"majrnq-36","timestamp":"2021-03-02T12:12:28.572857162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_IRyDbh-c9RlACKmHdE3Dkk2","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.600166_327.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.600166130Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.605568_328.1","section":"bucket.instance","name":"majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46","timestamp":"2021-03-02T12:12:28.605568203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_elrw1vOFDti0t3O82QLXXA3","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.607290_329.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.607290744Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.611670_330.1","section":"bucket","name":"majrnq-37","timestamp":"2021-03-02T12:12:28.611670219Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_gU3cBu3p9-SdkBGOKVmqX8n","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.654576_331.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.654576255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.660792_332.1","section":"bucket.instance","name":"majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47","timestamp":"2021-03-02T12:12:28.660792070Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_C4S4icmjkEj_KlHcsaSYOAA","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687148.662881_333.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.662881571Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"write"}}},{"id":"1_1614687148.669144_334.1","section":"bucket","name":"majrnq-38","timestamp":"2021-03-02T12:12:28.669144980Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UGPDwJFi_nG1cCQfKw_3S4V","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.493688_335.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.493688601Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.499525_337.1","section":"bucket.instance","name":"majrnq-39:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48","timestamp":"2021-03-02T12:12:35.499525622Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5nZsY7piy0cCwFJmtQdOiew","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.501571_338.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.501571886Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.506614_339.1","section":"bucket","name":"majrnq-39","timestamp":"2021-03-02T12:12:35.506614046Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YHnOt1jFTk4wXp7gmMD4s2w","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.535090_340.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.535090122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.539950_341.1","section":"bucket.instance","name":"majrnq-40:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49","timestamp":"2021-03-02T12:12:35.539950269Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_rhX4jZUlENWLJ41aWInepfH","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.541717_342.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.541717382Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.546130_343.1","section":"bucket","name":"majrnq-40","timestamp":"2021-03-02T12:12:35.546130857Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2-Z_eB3H621BD7pX3VatLZ7","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.589209_344.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.589209281Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.595705_345.1","section":"bucket.instance","name":"majrnq-41:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50","timestamp":"2021-03-02T12:12:35.595705087Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_HwOWcMA5In95l4orq8PJFAI","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687155.597587_346.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.597587656Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"write"}}},{"id":"1_1614687155.603950_347.1","section":"bucket","name":"majrnq-41","timestamp":"2021-03-02T12:12:35.603950195Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cNRWNu5Gsc9ZZp9kCHiFgQx","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:05.217 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:05.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:05.218 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:05.262 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:05.487 INFO:teuthology.orchestra.run.smithi012.stdout:[{"id":"1_1614687228.642810_208.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.642810424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.648951_209.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.648951861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.650881_210.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.650881563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.655713_211.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.655713697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.700863_212.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.700863566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.706796_213.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.706796999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.708467_214.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.708467524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.712852_215.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.712852708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:05.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:05.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:05.491 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:05.534 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:05.534 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:05.760 INFO:teuthology.orchestra.run.smithi036.stdout:[{"id":"1_1614687228.642810_208.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.642810424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.648951_209.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.648951861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.650881_210.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.650881563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.655713_211.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.655713697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.700863_212.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.700863566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.706796_213.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.706796999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.708467_214.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.708467524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.712852_215.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.712852708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:05.761 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:05.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:05.762 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:05.803 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:05.804 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:06.033 INFO:teuthology.orchestra.run.smithi012.stdout:[{"id":"1_1614687228.642810_208.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.642810424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.648951_209.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.648951861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.650881_210.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.650881563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.655713_211.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.655713697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.700863_212.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.700863566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.706796_213.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.706796999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.708467_214.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.708467524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.712852_215.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.712852708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:06.034 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.035 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.036 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:06.079 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:06.079 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:06.291 INFO:teuthology.orchestra.run.smithi036.stdout:[{"id":"1_1614687228.642810_208.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.642810424Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.648951_209.1","section":"bucket.instance","name":"majrnq-42:78024174-fc84-48b4-be73-8c69857e8755.5449.1","timestamp":"2021-03-02T12:13:48.648951861Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_V7sgLQitcLKe9vAbCZ4Tu44","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.650881_210.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.650881563Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.655713_211.1","section":"bucket","name":"majrnq-42","timestamp":"2021-03-02T12:13:48.655713697Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LSfL2vqNXdiU0B9wBzzcBaN","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.700863_212.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.700863566Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.706796_213.1","section":"bucket.instance","name":"majrnq-43:78024174-fc84-48b4-be73-8c69857e8755.5449.2","timestamp":"2021-03-02T12:13:48.706796999Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bKNoha7Grez6FoxMY9NNEKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687228.708467_214.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.708467524Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"write"}}},{"id":"1_1614687228.712852_215.1","section":"bucket","name":"majrnq-43","timestamp":"2021-03-02T12:13:48.712852708Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_XUa7Jq0mbatya2UbzOiwodX","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:06.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.293 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:06.342 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:06.342 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:06.570 INFO:teuthology.orchestra.run.smithi012.stdout:[{"id":"1_1614687295.063199_219.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.063199229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.067761_220.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.067761589Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.069271_221.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.069271246Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.073274_222.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.073274041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.113826_223.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.113826844Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.119064_224.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.119064811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.120739_225.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.120739567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.125219_226.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.125219067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:06.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.575 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:06.614 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:06.615 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:06.819 INFO:teuthology.orchestra.run.smithi036.stdout:[{"id":"1_1614687295.063199_219.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.063199229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.067761_220.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.067761589Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.069271_221.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.069271246Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.073274_222.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.073274041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.113826_223.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.113826844Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.119064_224.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.119064811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.120739_225.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.120739567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.125219_226.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.125219067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:06.820 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.820 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:06.822 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:06.862 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:06.863 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:07.087 INFO:teuthology.orchestra.run.smithi012.stdout:[{"id":"1_1614687295.063199_219.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.063199229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.067761_220.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.067761589Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.069271_221.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.069271246Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.073274_222.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.073274041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.113826_223.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.113826844Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.119064_224.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.119064811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.120739_225.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.120739567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.125219_226.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.125219067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:07.088 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:07.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:07.089 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:07.130 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:07.130 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:07.350 INFO:teuthology.orchestra.run.smithi036.stdout:[{"id":"1_1614687295.063199_219.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.063199229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.067761_220.1","section":"bucket.instance","name":"majrnq-44:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1","timestamp":"2021-03-02T12:14:55.067761589Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PuKL1zwtTuH0uBzU7--MU-Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.069271_221.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.069271246Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.073274_222.1","section":"bucket","name":"majrnq-44","timestamp":"2021-03-02T12:14:55.073274041Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zqs26UrUGGKIPfub_POKR67","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.113826_223.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.113826844Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.119064_224.1","section":"bucket.instance","name":"majrnq-45:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2","timestamp":"2021-03-02T12:14:55.119064811Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_bAUVdoBZCiWiOWc9-oyePkD","ver":1},"status":{"status":"complete"}}},{"id":"1_1614687295.120739_225.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.120739567Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"write"}}},{"id":"1_1614687295.125219_226.1","section":"bucket","name":"majrnq-45","timestamp":"2021-03-02T12:14:55.125219067Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_5BBPMnCabtyiN8Q4QXGYmQO","ver":1},"status":{"status":"complete"}}}] 2021-03-02T12:15:07.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:07.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:07.352 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:15:07.387 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:07.388 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:07.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:07.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:07.665 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:15:07.703 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:07.704 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:08.046 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:08.046 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:08.047 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:15:08.083 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:08.084 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:08.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:08.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:08.312 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:15:08.347 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:08.347 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:08.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:08.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:08.651 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:08.691 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:08.920 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:08.920 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:08.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:08.921 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:08.961 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:08.961 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:09.163 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:15:09.163 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.164 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:09.202 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:09.203 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:09.437 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:09.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '34dc26a2-30eb-4d82-a327-548c712f3e72', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.439 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 34dc26a2-30eb-4d82-a327-548c712f3e72 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:09.479 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:09.479 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:09.686 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:15:09.686 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.688 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:09.729 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:09.730 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:09.962 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:09.963 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.964 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:09.965 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:10.010 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:10.011 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:10.209 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:15:10.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.211 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:10.250 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:10.250 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:10.481 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:10.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'f31ea438-345b-4e8e-ac2d-a40213013c0a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.483 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period f31ea438-345b-4e8e-ac2d-a40213013c0a --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:10.521 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:10.522 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:10.734 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:15:10.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.735 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.736 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:10.775 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:10.990 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:10.991 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:10.992 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:11.032 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:11.033 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:11.243 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:15:11.243 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:11.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:11.244 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:11.282 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:11.283 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:11.502 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:11.502 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:11.503 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '133346b6-56ac-4f8e-91bd-8d51dd761892', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:11.504 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 133346b6-56ac-4f8e-91bd-8d51dd761892 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:11.542 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:11.543 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:11.741 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:15:11.888 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-03-02T12:15:11.901 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-46 2021-03-02T12:15:11.941 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-47 2021-03-02T12:15:12.001 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-48 2021-03-02T12:15:12.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:12.480 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:12.481 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:12.520 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:12.722 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687312.285418_184.1","last_update":"2021-03-02T12:15:12.285418Z"},{"marker":"1_1614687018.113106_165.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"1_1614687156.714816_207.1","last_update":"2021-03-02T12:12:36.714816Z"},{"marker":"1_1614686974.105927_93.3","last_update":"2021-03-02T12:09:34.105927Z"}] 2021-03-02T12:15:12.755 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:15:12.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:12.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:12.757 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:12.801 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:12.801 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:13.017 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687128.126601_141.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:08.126601Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:13.018 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687312.285418_184.1 target=1_1614687128.126601_141.6 2021-03-02T12:15:18.019 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.021 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:18.065 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:18.066 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:18.276 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:18.277 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:15:18.277 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.278 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.279 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:18.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:18.548 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687315.465002_191.1","last_update":"2021-03-02T12:15:15.465002Z"},{"marker":"1_1614687018.113106_165.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"1_1614687156.714816_207.1","last_update":"2021-03-02T12:12:36.714816Z"},{"marker":"1_1614686974.105927_93.3","last_update":"2021-03-02T12:09:34.105927Z"}] 2021-03-02T12:15:18.549 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:15:18.549 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.550 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.550 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:18.633 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:18.634 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:18.856 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7196637147885404157},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:18.857 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:15:18.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:18.859 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:18.936 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:19.155 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687315.465002_191.1","last_update":"2021-03-02T12:15:15.465002Z"},{"marker":"1_1614687018.113106_165.1","last_update":"2021-03-02T12:10:18.113106Z"},{"marker":"1_1614687156.714816_207.1","last_update":"2021-03-02T12:12:36.714816Z"},{"marker":"1_1614686974.105927_93.3","last_update":"2021-03-02T12:09:34.105927Z"}] 2021-03-02T12:15:19.156 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:15:19.156 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:19.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:19.157 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:19.202 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:19.203 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:19.474 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687018.113106_165.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:10:18.113106Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:19.474 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:15:19.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:19.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:19.476 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:19.521 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:19.521 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:19.735 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687313.605812_166.1","last_update":"2021-03-02T12:15:13.605812Z"},{"marker":"1_1614686997.049307_184.1","last_update":"2021-03-02T12:09:57.049307Z"},{"marker":"1_1614687156.702907_245.1","last_update":"2021-03-02T12:12:36.702907Z"},{"marker":"1_1614686975.042174_98.3","last_update":"2021-03-02T12:09:35.042174Z"}] 2021-03-02T12:15:19.736 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:15:19.737 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:19.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:19.738 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:19.781 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:19.782 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:20.016 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:20.017 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:25.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:25.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:25.030 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:25.073 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:25.074 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:25.292 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:25.321 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:30.323 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:30.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:30.325 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:30.369 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:30.370 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:30.597 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:30.598 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:35.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:35.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:35.602 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:35.644 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:35.645 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:35.871 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:36.059 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:41.061 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:41.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:41.077 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:41.122 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:41.359 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:41.360 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:46.362 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:46.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:46.364 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:46.407 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:46.638 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:46.643 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:51.644 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:51.655 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:51.655 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:51.698 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:51.698 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:51.926 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687129.061705_128.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:09.061705Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:51.927 WARNING:rgw_multi.tests:data of zone test-zone1 behind zone test-zone2: shard=0 master=1_1614687313.605812_166.1 target=1_1614687129.061705_128.6 2021-03-02T12:15:56.929 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:56.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:56.931 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:56.976 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:56.977 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:57.203 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687313.605812_166.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:13.605812Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:57.203 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:15:57.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:57.204 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:57.205 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:57.244 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:57.302 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:57.469 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687313.605812_166.1","last_update":"2021-03-02T12:15:13.605812Z"},{"marker":"1_1614686997.049307_184.1","last_update":"2021-03-02T12:09:57.049307Z"},{"marker":"1_1614687156.702907_245.1","last_update":"2021-03-02T12:12:36.702907Z"},{"marker":"1_1614686975.042174_98.3","last_update":"2021-03-02T12:09:35.042174Z"}] 2021-03-02T12:15:57.514 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:15:57.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:57.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:57.516 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:57.568 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:57.569 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:57.775 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13674358676509122085},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687313.605812_166.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:13.605812Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686997.049307_184.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:57.049307Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.702907_245.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.702907Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686975.042174_98.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:35.042174Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:57.854 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:15:57.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:57.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:57.856 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:57.895 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:57.895 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:58.121 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687313.610194_97.1","last_update":"2021-03-02T12:15:13.610194Z"},{"marker":"1_1614686996.099228_141.1","last_update":"2021-03-02T12:09:56.099228Z"},{"marker":"1_1614687156.425801_248.1","last_update":"2021-03-02T12:12:36.425801Z"},{"marker":"1_1614686974.092078_69.3","last_update":"2021-03-02T12:09:34.092078Z"}] 2021-03-02T12:15:58.121 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:15:58.122 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:58.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:58.123 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:58.203 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:58.346 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:58.438 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14630138824005222133},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687313.610194_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:13.610194Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686996.099228_141.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:56.099228Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.425801_248.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.425801Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.092078_69.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.092078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:58.612 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:15:58.612 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:58.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:58.613 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:58.654 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:58.766 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:58.874 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687313.610194_97.1","last_update":"2021-03-02T12:15:13.610194Z"},{"marker":"1_1614686996.099228_141.1","last_update":"2021-03-02T12:09:56.099228Z"},{"marker":"1_1614687156.425801_248.1","last_update":"2021-03-02T12:12:36.425801Z"},{"marker":"1_1614686974.092078_69.3","last_update":"2021-03-02T12:09:34.092078Z"}] 2021-03-02T12:15:58.874 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:15:58.875 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:58.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:58.876 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:58.919 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:59.001 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:59.145 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9850944892582908325},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687313.610194_97.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:13.610194Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614686996.099228_141.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:56.099228Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.425801_248.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.425801Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.092078_69.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.092078Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:15:59.232 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:15:59.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:59.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:59.239 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:15:59.275 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:59.276 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:59.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:59.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:15:59.511 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:15:59.604 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:59.606 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:15:59.834 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:15:59.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:59.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:15:59.835 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:15:59.879 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:15:59.882 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:16:00.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:16:00.098 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:16:00.099 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:16:00.178 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:16:00.178 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:16:00.394 INFO:teuthology.orchestra.run.smithi036.stdout:[] 2021-03-02T12:16:00.395 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:16:00.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:16:00.396 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:16:00.435 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:16:00.436 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:16:00.658 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:16:00.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:16:00.660 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:16:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:16:00.741 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:16:00.971 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:16:00.972 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_datalog_autotrim ... ok 2021-03-02T12:16:00.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:16:00.974 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:16:00.975 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:16:01.023 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:16:01.024 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:16:01.052 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:16:01.047+0000 7fb4b05e7080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:16:01.053 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:16:01.047+0000 7fb4b05e7080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:16:01.053 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:16:01.047+0000 7fb4b05e7080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:16:01.054 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:16:01.047+0000 7fb4b05e7080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:16:01.066 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:16:01.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:16:01.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:16:01.069 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:16:01.150 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:16:01.151 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:16:01.380 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:16:02.476 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":2,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:16:02.477 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-03-02T12:17:02.488 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-49 2021-03-02T12:17:02.579 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:02.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:02.580 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:02.618 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:02.805 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:17:02.837 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614687422.531259_289.1","last_update":"2021-03-02T12:17:02.531259Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:17:02.838 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:17:02.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:02.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:02.840 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:02.882 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:02.882 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:03.100 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:03.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:08.103 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:08.104 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:08.104 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:08.147 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:08.147 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:08.375 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:08.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:13.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:13.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:13.379 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:13.420 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:13.420 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:13.625 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:13.831 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:18.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:18.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:18.835 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:18.876 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:18.877 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:19.082 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:19.083 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:24.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:24.088 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:24.089 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:24.131 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:24.132 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:24.345 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:24.345 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:29.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:29.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:29.352 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:29.394 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:29.394 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:29.606 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:29.607 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:34.608 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:34.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:34.610 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:34.652 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:34.653 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:34.850 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687422.531259_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:17:02.531259Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:34.851 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:17:34.851 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:17:34.852 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:34.852 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:34.853 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:34.894 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:34.895 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:35.116 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:35.270 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:40.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:40.274 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:40.275 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:40.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:40.543 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:40.543 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:45.545 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:45.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:45.548 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:45.592 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:45.593 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:45.798 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:46.068 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:51.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:51.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:51.110 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:51.154 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:51.380 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:51.380 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:17:56.381 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:56.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:17:56.383 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:17:56.424 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:17:56.425 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:17:56.646 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:17:56.647 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:18:01.649 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:18:01.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:18:01.650 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:18:01.694 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:18:01.695 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:18:01.913 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687312.253753_256.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:12.253753Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:18:01.914 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687422.531259_289.1 target=1_1614687312.253753_256.1 2021-03-02T12:18:03.631 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:18:03.630+0000 7fef8e7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:18:03.632 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:18:03.625+0000 7f79c1ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:18:06.915 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:18:06.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:18:06.917 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:18:06.961 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:18:06.986 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:18:07.197 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687422.531259_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:17:02.531259Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:18:07.197 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:18:07.198 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:18:07.198 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:18:07.199 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:18:07.199 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:18:07.240 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:18:07.313 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:18:07.503 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687422.531259_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:17:02.531259Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:18:07.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:18:07.519 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=781413a3-49d9-4a3c-9ff5-00db01a70f6a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:18:07.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=781413a3-49d9-4a3c-9ff5-00db01a70f6a', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:18:07.520 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=781413a3-49d9-4a3c-9ff5-00db01a70f6a --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:18:07.586 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:18:07.648 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:18:07.649 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:18:07.609+0000 7f66b07b7080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:18:07.650 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:18:07.609+0000 7f66b07b7080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:18:07.650 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:18:07.609+0000 7f66b07b7080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:18:07.650 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:18:07.609+0000 7f66b07b7080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:18:07.651 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:18:07.652 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:18:07.652 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:18:07.653 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:18:07.734 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:18:07.735 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:18:07.960 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:18:09.056 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":3,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a"},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:18:09.057 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "781413a3-49d9-4a3c-9ff5-00db01a70f6a" 2021-03-02T12:19:09.398 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:19:09.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:19:09.402 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:19:09.443 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:19:09.444 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:19:09.473 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:19:09.468+0000 7f594d608080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:19:09.473 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:19:09.468+0000 7f594d608080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:19:09.474 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:19:09.468+0000 7f594d608080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:19:09.474 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:19:09.468+0000 7f594d608080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:19:09.485 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:19:09.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:19:09.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:19:09.487 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:19:09.566 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:19:09.566 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:19:09.784 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:19:10.871 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":4,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a"},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:19:10.872 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-03-02T12:20:10.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:20:10.876 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:20:10.876 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:20:10.918 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:20:10.918 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:20:10.948 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:20:10.943+0000 7f634a363080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:20:10.948 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:20:10.943+0000 7f634a363080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:20:10.949 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:20:10.943+0000 7f634a363080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:20:10.949 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:20:10.943+0000 7f634a363080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:20:10.960 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:20:10.961 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:20:10.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:20:10.962 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:20:11.041 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:20:11.042 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:20:11.274 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:20:12.361 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":5,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:20:12.362 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-03-02T12:21:12.364 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multi_zone_redirect ... ok 2021-03-02T12:21:12.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:12.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:12.370 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:12.411 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:12.411 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:12.588 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:21:12.621 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614687422.531259_289.1","last_update":"2021-03-02T12:17:02.531259Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:21:12.621 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:21:12.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:12.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:12.623 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:12.666 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:12.666 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:12.868 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687422.531259_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:17:02.531259Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:21:12.869 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:21:12.869 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:21:12.870 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:12.870 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:12.871 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:12.913 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:12.914 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:13.141 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687422.531259_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:17:02.531259Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:21:13.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:21:13.144 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:21:13.145 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:13.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:13.146 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:13.190 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:13.192 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:13.458 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687422.531259_289.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:17:02.531259Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:21:13.459 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:21:13.459 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:13.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', '7BHMFOIGOF04FEFE', '--secret', 'OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:13.462 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key 7BHMFOIGOF04FEFE --secret OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:13.541 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:13.541 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:13.600 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:13.594+0000 7fbf37678080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:21:13.601 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:13.594+0000 7fbf37678080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:21:13.602 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:13.594+0000 7fbf37678080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:21:13.603 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:13.594+0000 7fbf37678080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:21:13.612 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"ffc96321-d0c8-4c6f-85c3-0c571e4a965e","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"7BHMFOIGOF04FEFE","secret_key":"OjCNfg1DklvSFQFgy9RRfsA3BqGCQGOR"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:21:13.613 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:13.614 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:13.615 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:13.694 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:13.694 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:21.716 INFO:teuthology.orchestra.run.smithi036.stderr:Sending period to new master zone 781413a3-49d9-4a3c-9ff5-00db01a70f6a 2021-03-02T12:21:22.803 INFO:teuthology.orchestra.run.smithi036.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":6,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"ffc96321-d0c8-4c6f-85c3-0c571e4a965e","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055},{"key":"ffc96321-d0c8-4c6f-85c3-0c571e4a965e","val":1620392959}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:21:22.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:22.807 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:22.808 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:22.853 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:22.854 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:22.873 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:22.868+0000 7f0fe4b07080 0 failed reading obj info from .rgw.root:zone_info.0680c8ac-6fa2-460e-aca9-ec1ab695cedd: (2) No such file or directory 2021-03-02T12:21:22.874 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:22.868+0000 7f0fe4b07080 0 WARNING: could not read zone params for zone id=0680c8ac-6fa2-460e-aca9-ec1ab695cedd name=test-zone4 2021-03-02T12:21:22.874 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:22.868+0000 7f0fe4b07080 0 failed reading obj info from .rgw.root:zone_info.78024174-fc84-48b4-be73-8c69857e8755: (2) No such file or directory 2021-03-02T12:21:22.875 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:22.868+0000 7f0fe4b07080 0 WARNING: could not read zone params for zone id=78024174-fc84-48b4-be73-8c69857e8755 name=test-zone2 2021-03-02T12:21:22.883 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}} 2021-03-02T12:21:22.884 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:22.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:22.885 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:22.966 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:22.967 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:22.987 INFO:teuthology.orchestra.run.smithi012.stderr:zone name remove not found in zonegroup test-zonegroup 2021-03-02T12:21:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-03-02T12:21:22.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:22.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:22.995 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:23.033 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:23.034 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:23.090 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:23.082+0000 7f6684780080 0 failed reading obj info from .rgw.root:zone_info.781413a3-49d9-4a3c-9ff5-00db01a70f6a: (2) No such file or directory 2021-03-02T12:21:23.090 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:23.082+0000 7f6684780080 0 WARNING: could not read zone params for zone id=781413a3-49d9-4a3c-9ff5-00db01a70f6a name=test-zone1 2021-03-02T12:21:23.091 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:23.082+0000 7f6684780080 0 failed reading obj info from .rgw.root:zone_info.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78: (2) No such file or directory 2021-03-02T12:21:23.091 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:23.082+0000 7f6684780080 0 WARNING: could not read zone params for zone id=7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78 name=test-zone3 2021-03-02T12:21:23.094 INFO:teuthology.orchestra.run.smithi036.stderr:2021-03-02T12:21:23.086+0000 7f6684780080 0 zone id ffc96321-d0c8-4c6f-85c3-0c571e4a965e is not a part of zonegroup default 2021-03-02T12:21:23.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.110 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:23.152 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:23.219 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:23.216+0000 7f5fe80a7080 0 WARNING: can't generate connection for zone ffc96321-d0c8-4c6f-85c3-0c571e4a965e id remove: no endpoints defined 2021-03-02T12:21:23.452 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:21:23.448+0000 7fedf8730700 -1 failed to decode period 2021-03-02T12:21:23.453 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:21:23.446+0000 7f77eee6d700 -1 failed to decode period 2021-03-02T12:21:23.453 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:21:23.446+0000 7fb77d6c2700 -1 failed to decode period 2021-03-02T12:21:23.481 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":7,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:21:23.482 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zonegroup_remove ... ok 2021-03-02T12:21:23.484 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.485 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:23.558 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:23.559 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:23.596 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"a94ca0e6-6486-4f7d-99a6-9d94e4605248","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}} 2021-03-02T12:21:23.597 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.597 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.598 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:23.682 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:23.683 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:23.744 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"f8c474a6-f919-4bf7-9272-4b999409ded9","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69"} 2021-03-02T12:21:23.745 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:23.746 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:23.825 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:23.825 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:24.071 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:21:24.062+0000 7fb792eed700 -1 failed to decode period 2021-03-02T12:21:24.071 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:21:24.062+0000 7f77efe6f700 -1 failed to decode period 2021-03-02T12:21:24.111 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":8,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"a94ca0e6-6486-4f7d-99a6-9d94e4605248","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"f8c474a6-f919-4bf7-9272-4b999409ded9","zones":[{"id":"f8c474a6-f919-4bf7-9272-4b999409ded9","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055},{"key":"f8c474a6-f919-4bf7-9272-4b999409ded9","val":559631718}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:21:24.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.113 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:24.189 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:21:24.184+0000 7fedc5ecb700 -1 failed to decode period 2021-03-02T12:21:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:24.204 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:24.256+0000 7fde7e007080 0 zone id f8c474a6-f919-4bf7-9272-4b999409ded9 is not a part of zonegroup test-zonegroup 2021-03-02T12:21:24.259 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:24.256+0000 7fde7e007080 0 zone id f8c474a6-f919-4bf7-9272-4b999409ded9 is not a part of zonegroup default 2021-03-02T12:21:24.272 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.273 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:24.355 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:24.356 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:24.406 INFO:teuthology.orchestra.run.smithi012.stderr:2021-03-02T12:21:24.400+0000 7fb7c9b21080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-03-02T12:21:24.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-02T12:21:24.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.415 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:24.491 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:24.492 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:24.526 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:24.527 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:24.607 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:24.897 INFO:teuthology.orchestra.run.smithi012.stdout:{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","epoch":9,"predecessor_uuid":"f31ea438-345b-4e8e-ac2d-a40213013c0a","sync_status":["","1_1614687228.712852_215.1","",""],"period_map":{"id":"133346b6-56ac-4f8e-91bd-8d51dd761892","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","zones":[{"id":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","name":"test-zone4","endpoints":["http://smithi036.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"78024174-fc84-48b4-be73-8c69857e8755","name":"test-zone2","endpoints":["http://smithi036.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","name":"test-zone1","endpoints":["http://smithi012.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","name":"test-zone3","endpoints":["http://smithi012.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"0680c8ac-6fa2-460e-aca9-ec1ab695cedd","val":3474813242},{"key":"78024174-fc84-48b4-be73-8c69857e8755","val":3660196315},{"key":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","val":1025736551},{"key":"7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","val":2030736055}]},"master_zonegroup":"test-zonegroup","master_zone":"781413a3-49d9-4a3c-9ff5-00db01a70f6a","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"18323e0d-bcde-4a6a-931a-d01d6665ad69","realm_name":"test-realm","realm_epoch":4} 2021-03-02T12:21:24.898 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-03-02T12:21:24.900 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-50 2021-03-02T12:21:25.031 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:21:25.022+0000 7f77d9e43700 -1 failed to decode period 2021-03-02T12:21:25.032 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:21:25.022+0000 7fb7c4f51700 -1 failed to decode period 2021-03-02T12:21:25.032 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:21:25.028+0000 7fedcc6d8700 -1 failed to decode period 2021-03-02T12:21:25.086 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-51 2021-03-02T12:21:25.154 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-52 2021-03-02T12:21:25.526 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-53 2021-03-02T12:21:25.575 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-54 2021-03-02T12:21:25.657 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-55 2021-03-02T12:21:28.262 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_set_bucket_website ... tasks.rgw_multi.tests.test_set_bucket_policy ... ok 2021-03-02T12:21:28.263 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-56 2021-03-02T12:21:28.305 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-57 2021-03-02T12:21:28.373 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-58 2021-03-02T12:21:28.435 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:28.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-56', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:28.436 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-56 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:28.479 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:28.480 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:28.753 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:28.753 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-57', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:28.754 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-57 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:28.836 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:28.892 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:29.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:29.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-58', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:21:29.133 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-58 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:21:29.216 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:29.258 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:29.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:29.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:29.664 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:29.704 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:29.705 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:29.965 INFO:teuthology.orchestra.run.smithi012.stdout:["obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.78024174-fc84-48b4-be73-8c69857e8755","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.0","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.0.retry","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","obj_delete_at_hint.0000000026","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","obj_delete_at_hint.0000000028","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.2","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28-majrnq-19","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","sync.error-log.6","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.6-majrnq-55","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20-majrnq-11","obj_delete_at_hint.0000000015","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37-majrnq-28","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","meta.log.133346b6-56ac-4f8e-91bd-8d51dd761892.1","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","data_log.3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1-majrnq-49","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1-majrnq-1","obj_delete_at_hint.0000000096","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49-majrnq-40","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4-majrnq-4","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.0.retry","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26-majrnq-17","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","bilog.trim","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.1","obj_delete_at_hint.0000000016","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40-majrnq-31","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.36-majrnq-27","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","sync.error-log.4","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","obj_delete_at_hint.0000000014","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18-majrnq-9","obj_delete_at_hint.0000000064","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.1-majrnq-50","obj_delete_at_hint.0000000071","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27-majrnq-18","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","obj_delete_at_hint.0000000074","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34-majrnq-25","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29-majrnq-20","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.2-majrnq-51","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44-majrnq-35","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6-majrnq-6","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2-majrnq-45","obj_delete_at_hint.0000000077","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19-majrnq-10","obj_delete_at_hint.0000000032","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.42-majrnq-33","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21-majrnq-12","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38-majrnq-29","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.1","sync.error-log.3","datalog.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","mdlog.sync-status.shard.2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.2","obj_delete_at_hint.0000000045","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","obj_delete_at_hint.0000000002","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.39-majrnq-30","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23-majrnq-14","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.5-majrnq-54","obj_delete_at_hint.0000000116","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","obj_delete_at_hint.0000000034","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.3","obj_delete_at_hint.0000000101","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24-majrnq-15","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","mdlog.sync-status.shard.1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.2.retry","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.7-majrnq-56","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.9-majrnq-58","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","obj_delete_at_hint.0000000036","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1-majrnq-44","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22-majrnq-13","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","obj_delete_at_hint.0000000039","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46-majrnq-37","obj_delete_at_hint.0000000100","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.4-majrnq-53","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","obj_delete_at_hint.0000000093","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.2","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","sync.error-log.2","data_log.0","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.0","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31-majrnq-22","obj_delete_at_hint.0000000012","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33-majrnq-24","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4-majrnq-47","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2-majrnq-2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","obj_delete_at_hint.0000000072","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","obj_delete_at_hint.0000000062","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","obj_delete_at_hint.0000000115","sync.error-log.1","obj_delete_at_hint.0000000033","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","obj_delete_at_hint.0000000025","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.8-majrnq-57","obj_delete_at_hint.0000000123","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.1.retry","obj_delete_at_hint.0000000108","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45-majrnq-36","obj_delete_at_hint.0000000094","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5-majrnq-48","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.0","obj_delete_at_hint.0000000013","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.2","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48-majrnq-39","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25-majrnq-16","2021-03-02-12-78024174-fc84-48b4-be73-8c69857e8755.5449.1-majrnq-42","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","2021-03-02-12-78024174-fc84-48b4-be73-8c69857e8755.5449.2-majrnq-43","obj_delete_at_hint.0000000023","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.0","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.1.retry","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3-majrnq-3","obj_delete_at_hint.0000000107","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","obj_delete_at_hint.0000000046","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.3.retry","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.1","meta.full-sync.index.2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50-majrnq-41","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35-majrnq-26","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","obj_delete_at_hint.0000000086","sync.error-log.5","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30-majrnq-21","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17-majrnq-8","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.2.retry","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41-majrnq-32","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.3-majrnq-52","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5-majrnq-5","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47-majrnq-38","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","obj_delete_at_hint.0000000050","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000102","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16-majrnq-7","obj_delete_at_hint.0000000124","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","obj_delete_at_hint.0000000075","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3-majrnq-46","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.3.retry","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32-majrnq-23","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43-majrnq-34"] 2021-03-02T12:21:29.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:29.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:29.995 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:30.035 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:30.036 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:30.255 INFO:teuthology.orchestra.run.smithi036.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.0.retry","obj_delete_at_hint.0000000026","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","obj_delete_at_hint.0000000028","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28-majrnq-19","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20-majrnq-11","obj_delete_at_hint.0000000015","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.37-majrnq-28","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","meta.log.133346b6-56ac-4f8e-91bd-8d51dd761892.1","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","data_log.3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1-majrnq-49","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1-majrnq-1","obj_delete_at_hint.0000000096","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49-majrnq-40","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4-majrnq-4","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26-majrnq-17","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","bilog.trim","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.1","obj_delete_at_hint.0000000016","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.40-majrnq-31","obj_delete_at_hint.0000000041","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2.retry","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3.retry","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000014","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18-majrnq-9","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27-majrnq-18","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","obj_delete_at_hint.0000000074","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34-majrnq-25","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29-majrnq-20","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.2-majrnq-51","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6-majrnq-6","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2-majrnq-45","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000077","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19-majrnq-10","obj_delete_at_hint.0000000032","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21-majrnq-12","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","datalog.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","obj_delete_at_hint.0000000002","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23-majrnq-14","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.5-majrnq-54","obj_delete_at_hint.0000000116","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","obj_delete_at_hint.0000000034","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.3","obj_delete_at_hint.0000000101","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24-majrnq-15","mdlog.sync-status.shard.1","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.2.retry","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","obj_delete_at_hint.0000000073","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","obj_delete_at_hint.0000000038","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000119","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1-majrnq-44","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22-majrnq-13","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","obj_delete_at_hint.0000000039","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46-majrnq-37","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.2","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.0","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31-majrnq-22","obj_delete_at_hint.0000000012","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33-majrnq-24","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0.retry","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","obj_delete_at_hint.0000000020","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4-majrnq-47","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2-majrnq-2","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","obj_delete_at_hint.0000000025","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.8-majrnq-57","obj_delete_at_hint.0000000123","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.1.retry","obj_delete_at_hint.0000000108","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45-majrnq-36","obj_delete_at_hint.0000000094","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5-majrnq-48","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.0","obj_delete_at_hint.0000000013","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.2","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48-majrnq-39","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25-majrnq-16","2021-03-02-12-78024174-fc84-48b4-be73-8c69857e8755.5449.1-majrnq-42","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","2021-03-02-12-78024174-fc84-48b4-be73-8c69857e8755.5449.2-majrnq-43","obj_delete_at_hint.0000000023","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3-majrnq-3","obj_delete_at_hint.0000000107","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","obj_delete_at_hint.0000000046","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.1","meta.full-sync.index.2","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","data_log.2","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","obj_delete_at_hint.0000000067","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50-majrnq-41","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35-majrnq-26","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","obj_delete_at_hint.0000000086","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","data.full-sync.index.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30-majrnq-21","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17-majrnq-8","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1.retry","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5-majrnq-5","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47-majrnq-38","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","datalog.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a","obj_delete_at_hint.0000000050","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000102","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16-majrnq-7","obj_delete_at_hint.0000000124","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","bucket.sync-status.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1","obj_delete_at_hint.0000000075","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3-majrnq-46","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","datalog.sync-status.shard.7911a4b1-c8a3-4af5-9b1e-a2c87edb3e78.3.retry","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32-majrnq-23","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.43-majrnq-34"] 2021-03-02T12:21:30.284 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:30.285 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:30.285 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:30.326 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:30.595 INFO:teuthology.orchestra.run.smithi012.stdout:["obj_delete_at_hint.0000000078","meta.history","datalog.sync-status.78024174-fc84-48b4-be73-8c69857e8755","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.0","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","obj_delete_at_hint.0000000026","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","obj_delete_at_hint.0000000028","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28-majrnq-19","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.6-majrnq-55","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20-majrnq-11","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","meta.log.133346b6-56ac-4f8e-91bd-8d51dd761892.1","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","data_log.3","meta.log.f31ea438-345b-4e8e-ac2d-a40213013c0a.1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.3","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1-majrnq-49","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.1-majrnq-1","obj_delete_at_hint.0000000096","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.49-majrnq-40","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.4-majrnq-4","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.0.retry","meta.log.34dc26a2-30eb-4d82-a327-548c712f3e72.0","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26-majrnq-17","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000041","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2.retry","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3.retry","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000014","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.18-majrnq-9","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27-majrnq-18","obj_delete_at_hint.0000000074","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34-majrnq-25","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29-majrnq-20","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.44-majrnq-35","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.6-majrnq-6","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.2-majrnq-45","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000077","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19-majrnq-10","obj_delete_at_hint.0000000032","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.3","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21-majrnq-12","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.38-majrnq-29","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","mdlog.sync-status.shard.2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.2","obj_delete_at_hint.0000000045","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","obj_delete_at_hint.0000000002","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23-majrnq-14","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","obj_delete_at_hint.0000000116","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24-majrnq-15","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","mdlog.sync-status.shard.1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","obj_delete_at_hint.0000000073","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","obj_delete_at_hint.0000000038","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.9-majrnq-58","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.1-majrnq-44","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22-majrnq-13","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","obj_delete_at_hint.0000000039","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46-majrnq-37","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","obj_delete_at_hint.0000000100","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31-majrnq-22","obj_delete_at_hint.0000000012","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33-majrnq-24","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0.retry","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","obj_delete_at_hint.0000000020","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4-majrnq-47","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.2-majrnq-2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","obj_delete_at_hint.0000000072","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","obj_delete_at_hint.0000000062","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","obj_delete_at_hint.0000000108","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45-majrnq-36","obj_delete_at_hint.0000000094","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5-majrnq-48","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.48-majrnq-39","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25-majrnq-16","2021-03-02-12-78024174-fc84-48b4-be73-8c69857e8755.5449.1-majrnq-42","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3","meta.log.34dc26a2-30eb-4d82-a327-548c712f3e72.1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","2021-03-02-12-78024174-fc84-48b4-be73-8c69857e8755.5449.2-majrnq-43","obj_delete_at_hint.0000000023","data.full-sync.index.78024174-fc84-48b4-be73-8c69857e8755.0","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.1.retry","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.3-majrnq-3","obj_delete_at_hint.0000000107","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","obj_delete_at_hint.0000000046","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.3.retry","meta.full-sync.index.2","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","data_log.2","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","obj_delete_at_hint.0000000067","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.50-majrnq-41","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35-majrnq-26","obj_delete_at_hint.0000000086","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30-majrnq-21","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.17-majrnq-8","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1.retry","datalog.sync-status.shard.78024174-fc84-48b4-be73-8c69857e8755.2.retry","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.41-majrnq-32","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.3-majrnq-52","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.5-majrnq-5","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47-majrnq-38","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","datalog.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.16-majrnq-7","obj_delete_at_hint.0000000124","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.78024174-fc84-48b4-be73-8c69857e8755:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1","obj_delete_at_hint.0000000075","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3-majrnq-46","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","2021-03-02-12-781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32-majrnq-23"] 2021-03-02T12:21:30.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:30.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone4', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:30.623 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone4 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:30.665 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:30.666 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:30.887 INFO:teuthology.orchestra.run.smithi036.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-20:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.29:8","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-15:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.24:1","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","obj_delete_at_hint.0000000024","meta.log.133346b6-56ac-4f8e-91bd-8d51dd761892.1","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","meta.log.f31ea438-345b-4e8e-ac2d-a40213013c0a.1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:5","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","meta.log.34dc26a2-30eb-4d82-a327-548c712f3e72.0","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:5","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","obj_delete_at_hint.0000000041","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2.retry","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3.retry","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-13:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.22:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:3","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-37:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.46:10","obj_delete_at_hint.0000000014","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-26:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.35:8","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:3","obj_delete_at_hint.0000000077","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1","obj_delete_at_hint.0000000032","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:3","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-16:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.25:1","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-38:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.47:10","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-25:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.34:8","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-46:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.3:8","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-22:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.31:8","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-24:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.33:8","obj_delete_at_hint.0000000066","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:3","obj_delete_at_hint.0000000038","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-49:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6082.1:5","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-21:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.30:8","obj_delete_at_hint.0000000039","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:7","obj_delete_at_hint.0000000100","obj_delete_at_hint.0000000093","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:3","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","obj_delete_at_hint.0000000012","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0.retry","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-47:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.4:8","obj_delete_at_hint.0000000020","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:8","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-11:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.20:7","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:7","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-12:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.21:1","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.3","meta.log.34dc26a2-30eb-4d82-a327-548c712f3e72.1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-19:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.28:8","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.0","obj_delete_at_hint.0000000107","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-48:781413a3-49d9-4a3c-9ff5-00db01a70f6a.5851.5:8","obj_delete_at_hint.0000000046","meta.full-sync.index.2","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:5","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-36:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.45:10","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-18:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.27:3","obj_delete_at_hint.0000000067","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.2","obj_delete_at_hint.0000000086","datalog.sync-status.shard.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1.retry","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-17:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.26:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-14:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.23:1","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-23:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.32:8","datalog.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","bucket.sync-status.781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-10:781413a3-49d9-4a3c-9ff5-00db01a70f6a.4344.19:8","data.full-sync.index.781413a3-49d9-4a3c-9ff5-00db01a70f6a.1","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030"] 2021-03-02T12:21:30.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:30.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:30.936 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:30.979 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:30.980 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:31.206 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687689.468960_341.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_560.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_328.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_408.2","last_update":"2021-03-02T12:21:29.468960Z"}] 2021-03-02T12:21:31.207 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:21:31.207 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:31.208 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:31.208 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:31.255 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:31.256 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:31.472 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:21:31.473 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:21:36.474 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:36.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:36.476 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:36.520 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:36.520 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:36.720 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:21:36.721 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:21:41.723 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:41.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:41.724 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:41.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:41.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:41.986 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:21:41.987 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:21:46.988 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:46.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:46.989 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:47.034 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:47.034 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:47.254 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:21:47.254 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:21:52.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:52.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:52.257 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:52.303 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:52.303 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:52.524 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:21:52.524 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:21:57.526 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:57.527 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:21:57.528 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:21:57.573 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:21:57.574 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:21:57.798 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:21:57.799 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:02.801 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:02.802 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:02.802 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:02.849 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:02.849 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:03.067 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:03.068 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:08.068 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:08.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:08.071 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:08.115 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:08.116 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:08.333 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:08.334 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:13.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:13.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:13.337 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:13.385 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:13.386 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:13.622 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:13.622 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:18.624 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:18.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:18.625 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:18.669 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:18.670 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:18.886 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:18.887 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:23.888 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:23.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:23.889 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:23.935 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:23.935 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:24.160 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:24.161 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:29.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:29.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:29.166 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:29.211 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:29.211 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:29.431 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_560.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_328.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_408.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:29.432 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:22:29.433 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:29.435 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:29.436 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:29.476 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:29.477 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:29.703 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687689.468960_341.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_560.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_328.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_408.2","last_update":"2021-03-02T12:21:29.468960Z"}] 2021-03-02T12:22:29.745 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:22:29.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:29.746 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:29.747 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:29.791 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:30.015 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7196637147885404157},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_560.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_328.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_408.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:30.016 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:22:30.016 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:30.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:30.017 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:30.097 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:30.196 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:30.319 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687689.468960_341.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_560.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_328.3","last_update":"2021-03-02T12:21:29.468960Z"},{"marker":"1_1614687689.468960_408.2","last_update":"2021-03-02T12:21:29.468960Z"}] 2021-03-02T12:22:30.455 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:22:30.456 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:30.457 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:30.458 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:30.502 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:30.503 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:30.766 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:30.767 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:35.769 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:35.770 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:35.771 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:35.817 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:35.817 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:36.081 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:36.082 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:41.083 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:41.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:41.302 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:41.351 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:41.352 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:41.621 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:41.713 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:46.714 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:46.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:46.716 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:46.756 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:46.757 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:47.025 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:47.025 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:52.027 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:52.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:52.034 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:52.078 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:52.078 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:52.344 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:52.345 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:22:57.346 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:57.347 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:22:57.348 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:22:57.391 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:22:57.391 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:22:57.657 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:22:57.658 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:23:02.660 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:02.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:02.663 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:02.712 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:02.712 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:02.967 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:02.967 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:23:07.969 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:07.971 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:07.972 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:08.015 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:08.016 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:08.283 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:08.283 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:23:13.285 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:13.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:13.287 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:13.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:13.842 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:13.843 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:13.844 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:23:18.846 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:18.847 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:18.847 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:18.892 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:18.892 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:19.159 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:19.160 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:23:24.162 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:24.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:24.179 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:24.225 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:24.225 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:24.492 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687315.465002_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:15:15.465002Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687550.910901_455.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:19:10.910901Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687156.714816_207.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:12:36.714816Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614686974.105927_93.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:09:34.105927Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:24.569 WARNING:rgw_multi.tests:data of zone test-zone4 behind zone test-zone1: shard=0 master=1_1614687689.468960_341.3 target=1_1614687315.465002_191.1, shard=1 master=1_1614687689.468960_560.3 target=1_1614687550.910901_455.1, shard=2 master=1_1614687689.468960_328.3 target=1_1614687156.714816_207.1, shard=3 master=1_1614687689.468960_408.2 target=1_1614686974.105927_93.3 2021-03-02T12:23:29.571 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:29.572 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:29.572 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:29.615 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:29.615 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:29.881 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_341.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_560.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_328.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687689.468960_408.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:29.468960Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:29.882 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:23:29.883 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:29.884 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:29.885 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:29.962 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:29.963 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:30.170 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687709.020343_266.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687709.020343_566.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687709.020343_382.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687709.020343_330.2","last_update":"2021-03-02T12:21:49.020343Z"}] 2021-03-02T12:23:30.171 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:23:30.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.172 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:30.214 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:30.215 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:30.451 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_566.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_382.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_330.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:30.451 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:23:30.452 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.453 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:30.492 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:30.493 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:30.702 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687709.020343_266.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687709.020343_566.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687709.020343_382.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687709.020343_330.2","last_update":"2021-03-02T12:21:49.020343Z"}] 2021-03-02T12:23:30.703 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:23:30.703 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.704 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:30.748 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:30.749 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:30.972 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13674358676509122085},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_266.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_566.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_382.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_330.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:30.972 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:23:30.973 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:30.974 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:31.059 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:31.060 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:31.284 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687751.312219_171.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687751.312219_502.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687751.312219_365.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687751.312219_242.2","last_update":"2021-03-02T12:22:31.312219Z"}] 2021-03-02T12:23:31.285 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:23:31.285 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:31.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:31.287 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:31.369 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:31.370 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:31.593 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14630138824005222133},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_171.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_502.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_365.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_242.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:31.594 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:23:31.594 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:31.595 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:31.595 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:31.673 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:31.674 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:31.888 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687751.312219_171.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687751.312219_502.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687751.312219_365.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687751.312219_242.2","last_update":"2021-03-02T12:22:31.312219Z"}] 2021-03-02T12:23:31.889 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:23:31.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:31.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:31.890 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:31.935 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:31.935 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:32.143 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9850944892582908325},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_171.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_502.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_365.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_242.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:23:32.144 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:23:32.145 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable ... ok 2021-03-02T12:23:32.148 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-59 2021-03-02T12:23:32.191 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-60 2021-03-02T12:23:32.259 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-61 2021-03-02T12:23:32.923 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:32.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:32.924 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:32.965 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:32.965 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:33.191 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:33.192 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:23:33.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:33.193 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:33.194 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:33.239 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:33.239 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:33.462 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:33.463 WARNING:rgw_multi.tests:bucket majrnq-59 zone test-zone2 behind zone test-zone1: shard=0 master=00000000001.40.5 target=, shard=2 master=00000000001.62.5 target=, shard=4 master=00000000001.49.5 target=, shard=7 master=00000000001.64.5 target= 2021-03-02T12:23:38.463 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:38.464 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:38.465 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:38.518 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:38.519 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:38.755 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:38.755 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:23:38.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:38.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:38.757 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:38.801 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:38.802 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:39.039 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:39.040 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:23:39.040 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:39.041 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:39.041 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:39.126 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:39.384 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:39.385 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:23:39.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:39.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:39.387 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:39.466 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:39.467 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:39.703 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.62.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.49.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.64.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:39.703 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:23:39.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:39.704 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:39.705 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:39.750 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:39.750 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:40.014 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.62.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:40.015 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:23:40.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.017 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:40.096 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:40.096 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:40.321 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.30.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:40.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:23:40.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.331 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:40.376 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:40.376 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:40.624 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:40.625 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:23:40.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.627 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:40.670 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:40.671 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:40.888 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.30.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.43.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.62.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:40.889 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:23:40.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:40.890 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:40.935 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:40.936 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:41.176 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:41.177 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:23:41.178 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:41.178 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:41.179 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:41.220 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:41.221 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:41.454 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:41.454 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:23:41.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:41.455 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:41.456 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:41.540 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:41.541 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:41.783 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:41.783 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:23:41.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:41.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:41.785 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:41.827 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:41.828 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:42.049 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.64.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.85.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:42.049 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:23:42.050 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:42.050 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:42.051 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:42.097 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:42.098 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:42.327 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:42.328 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:23:42.329 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-59 zones={test-zone2, test-zone1} 2021-03-02T12:23:42.344 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:42.352 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:42.353 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:42.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:42.354 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:42.354 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:42.362 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:42.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:42.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:42.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:42.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:42.432 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:42.545 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:42.654 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:42.712 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:42.829 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:42.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:42.999 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:43.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:43.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:43.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:43.393 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:43.504 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-59 zones={test-zone2, test-zone1} 2021-03-02T12:23:43.505 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-59 zones={test-zone3, test-zone1} 2021-03-02T12:23:43.514 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:43.521 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:43.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:43.522 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:43.523 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:43.523 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:43.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:43.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:43.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:43.531 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:43.548 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:43.602 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:43.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:43.865 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:43.923 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:44.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:44.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:44.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:44.328 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:44.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:44.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:44.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:44.931 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-59 zones={test-zone3, test-zone1} 2021-03-02T12:23:45.194 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-59 zones={test-zone3, test-zone2} 2021-03-02T12:23:45.210 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:45.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:45.220 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:45.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:45.221 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:45.222 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:45.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:45.234 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:45.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:45.235 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:45.295 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:45.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:45.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:45.788 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:45.848 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:45.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:46.243 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:46.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:46.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:46.569 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:46.630 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:46.746 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:47.047 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-59 zones={test-zone3, test-zone2} 2021-03-02T12:23:47.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:47.048 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:47.049 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:47.089 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:47.090 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:47.320 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.31.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:47.438 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:23:47.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:47.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:47.439 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:47.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:47.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:47.742 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:47.743 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:23:47.744 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:47.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:47.745 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:47.793 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:47.793 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:48.033 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.31.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:48.033 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:23:48.034 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:48.034 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:48.035 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:48.125 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:48.363 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.31.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:48.364 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:23:48.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:48.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:48.365 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:48.448 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:48.449 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:48.667 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.64.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.31.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:48.668 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:23:48.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:48.669 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:48.670 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:48.716 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:48.716 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:49.078 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.64.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.31.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:49.079 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:23:49.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.080 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:49.169 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:49.170 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:49.393 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.39.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.30.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:49.394 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:23:49.394 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.396 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:49.442 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:49.442 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:49.688 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:49.688 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:23:49.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.689 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.690 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:49.733 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:49.733 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:49.951 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.72.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.39.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.30.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:49.951 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:23:49.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:49.953 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:49.997 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:49.997 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:50.237 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:50.237 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:23:50.238 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:50.238 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:50.239 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:50.318 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:50.554 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.66.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.32.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:50.555 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:23:50.556 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:50.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:50.557 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:50.641 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:50.642 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:50.895 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:50.896 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:23:50.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:50.897 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:50.898 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:50.980 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:50.981 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:51.212 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.66.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.32.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:51.213 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:23:51.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:51.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:51.214 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:51.257 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:51.258 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:51.486 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.32.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:51.486 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:23:51.487 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-60 zones={test-zone2, test-zone1} 2021-03-02T12:23:51.498 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:51.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:51.504 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:51.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:51.505 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:51.505 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:51.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:51.515 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:51.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:51.516 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:51.530 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:51.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:51.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:51.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:51.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:51.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:52.074 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:52.127 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:52.242 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:52.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:52.407 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:52.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:52.639 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-60 zones={test-zone2, test-zone1} 2021-03-02T12:23:52.640 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-60 zones={test-zone3, test-zone1} 2021-03-02T12:23:52.650 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:52.657 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:52.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:52.658 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:52.659 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:52.660 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:52.666 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:52.667 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:52.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:52.668 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:52.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:52.736 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:52.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:52.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:53.003 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:53.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:53.222 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:53.275 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:53.396 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:53.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:53.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:53.680 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:53.787 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-60 zones={test-zone3, test-zone1} 2021-03-02T12:23:53.788 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-60 zones={test-zone3, test-zone2} 2021-03-02T12:23:53.799 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:53.805 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:53.806 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:53.807 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:53.808 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:53.809 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:53.816 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:53.817 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:53.818 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:53.819 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:53.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:53.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:54.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:54.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:54.191 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:54.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:54.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:54.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:54.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:23:54.698 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:54.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:54.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:23:54.973 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-60 zones={test-zone3, test-zone2} 2021-03-02T12:23:54.974 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:54.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:54.975 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:55.018 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:55.252 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.75.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:55.253 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:23:55.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:55.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:55.254 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:55.302 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:55.302 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:55.536 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:55.536 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:23:55.537 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:55.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:55.538 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:55.581 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:55.581 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:55.808 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.75.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:55.810 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:23:55.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:55.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:55.812 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:55.896 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:55.897 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:56.148 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:56.251 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:23:56.252 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:56.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:56.254 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:56.297 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:56.519 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.75.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.50.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:56.520 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:23:56.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:56.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:56.521 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:56.569 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:56.570 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:56.834 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.75.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:56.835 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:23:56.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:56.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:56.840 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:56.915 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:56.916 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:57.146 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:57.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:23:57.149 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:57.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:57.150 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:57.197 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:57.198 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:57.436 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:57.437 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:23:57.437 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:57.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:57.438 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:57.485 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:57.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:57.723 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.43.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.76.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:57.745 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:23:57.746 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:57.747 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:57.748 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:57.794 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:57.795 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:58.027 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.76.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:58.028 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:23:58.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.029 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:58.111 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:58.111 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:58.338 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.80.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.72.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.48.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:58.339 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:23:58.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.341 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:58.435 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:58.435 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:58.658 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:58.659 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:23:58.660 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.661 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.662 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:58.747 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:58.747 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:58.976 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.80.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.72.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.48.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:23:58.977 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:23:58.977 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.978 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:23:58.978 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:23:59.048 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:23:59.051 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:23:59.272 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.80.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.72.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:23:59.274 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:23:59.275 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-61 zones={test-zone2, test-zone1} 2021-03-02T12:23:59.298 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:23:59.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:59.313 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:59.319 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:59.320 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:23:59.337 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:23:59.338 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:23:59.345 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:23:59.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:23:59.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:59.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:59.652 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:23:59.790 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:23:59.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:00.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:00.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:00.193 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:00.317 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:00.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:00.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:00.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:00.712 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-61 zones={test-zone2, test-zone1} 2021-03-02T12:24:00.714 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-61 zones={test-zone3, test-zone1} 2021-03-02T12:24:00.724 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:00.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:00.735 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:00.736 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:00.736 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:00.742 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:00.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:00.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:00.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:00.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:00.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:00.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:01.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:01.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:01.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:01.323 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:01.381 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:01.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:01.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:01.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:01.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:01.875 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-61 zones={test-zone3, test-zone1} 2021-03-02T12:24:01.876 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-61 zones={test-zone3, test-zone2} 2021-03-02T12:24:01.885 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:01.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:01.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:01.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:01.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:01.893 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:01.899 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:01.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:01.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:01.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:01.914 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:01.970 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:02.077 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:02.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:02.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:02.352 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:02.457 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:02.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:02.637 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:02.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:02.800 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:02.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:03.026 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-61 zones={test-zone3, test-zone2} 2021-03-02T12:24:03.028 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:03.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:03.030 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:24:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:03.078 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:03.364 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'majrnq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:03.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'majrnq-59', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:03.366 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket majrnq-59 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:24:03.449 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:03.450 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:03.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:03.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:03.736 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:24:03.819 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:03.820 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:04.112 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'majrnq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:04.113 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'majrnq-60', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:04.113 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket majrnq-60 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:24:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:04.194 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:04.482 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:04.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:04.484 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:24:04.567 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:04.568 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:04.848 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'majrnq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:04.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'majrnq-61', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:24:04.849 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket majrnq-61 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:24:04.938 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:04.938 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:05.864 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:05.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:05.866 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:05.914 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:06.148 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.46.5"},{"key":1,"val":"00000000004.94.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.61.5"},{"key":5,"val":"00000000004.41.1"},{"key":6,"val":"00000000004.42.1"},{"key":7,"val":"00000000006.74.5"},{"key":8,"val":"00000000004.42.1"},{"key":9,"val":"00000000004.55.1"},{"key":10,"val":"00000000004.53.1"}]} 2021-03-02T12:24:06.148 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:24:06.149 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:06.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:06.150 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:06.200 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:06.200 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:06.442 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"full-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"full-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.61.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":8,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-03-02T12:24:06.443 WARNING:rgw_multi.tests:bucket majrnq-59 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.46.5 target=00000000001.40.5, shard=1 master=00000000004.94.1 target=, shard=3 master=00000000004.54.1 target=, shard=5 master=00000000004.41.1 target=, shard=7 master=00000000006.74.5 target=00000000001.64.5, shard=9 master=00000000004.55.1 target= 2021-03-02T12:24:11.445 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:11.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:11.446 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:11.492 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:11.492 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:11.714 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.46.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.94.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.61.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-03-02T12:24:11.714 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:24:11.715 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:11.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:11.716 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:11.757 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:11.758 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:11.997 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.46.5"},{"key":1,"val":"00000000004.94.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.61.5"},{"key":5,"val":"00000000004.41.1"},{"key":6,"val":"00000000004.42.1"},{"key":7,"val":"00000000006.74.5"},{"key":8,"val":"00000000004.42.1"},{"key":9,"val":"00000000004.55.1"},{"key":10,"val":"00000000004.53.1"}]} 2021-03-02T12:24:11.998 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:24:11.999 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.003 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:12.082 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:12.083 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:12.320 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.46.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.94.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.61.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.74.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-03-02T12:24:12.321 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:24:12.322 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.324 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:12.406 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:12.407 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:12.652 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.46.5"},{"key":1,"val":"00000000004.94.1"},{"key":2,"val":"00000000006.74.5"},{"key":3,"val":"00000000004.54.1"},{"key":4,"val":"00000000006.61.5"},{"key":5,"val":"00000000004.41.1"},{"key":6,"val":"00000000004.42.1"},{"key":7,"val":"00000000006.74.5"},{"key":8,"val":"00000000004.42.1"},{"key":9,"val":"00000000004.55.1"},{"key":10,"val":"00000000004.53.1"}]} 2021-03-02T12:24:12.652 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:24:12.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.654 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:12.700 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:12.701 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:12.964 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.94.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.54.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.61.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.41.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.74.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.53.1"}}}] 2021-03-02T12:24:12.964 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-59 2021-03-02T12:24:12.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:12.966 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:13.009 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:13.009 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:13.233 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000002.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.84.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.68.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:13.233 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:24:13.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:13.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:13.235 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:13.283 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:13.284 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:13.533 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.84.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:13.533 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:24:13.534 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:13.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:13.535 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:13.580 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:13.580 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:13.792 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000002.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.84.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.51.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.68.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:13.793 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:24:13.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:13.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:13.794 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:13.841 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:13.841 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:14.082 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.84.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:14.082 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-59 2021-03-02T12:24:14.083 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.084 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:14.125 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:14.125 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:14.363 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000002.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.70.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.87.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:14.364 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:24:14.365 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.365 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.366 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:14.450 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:14.451 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:14.689 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.70.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.87.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:14.690 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:24:14.690 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-59', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.691 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-59 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:14.731 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:14.732 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:14.959 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000002.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.70.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.46.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.87.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:14.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:24:14.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-59', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:14.961 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-59 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:15.005 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:15.005 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:15.238 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.70.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.87.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:15.239 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-59 2021-03-02T12:24:15.240 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-59 zones={test-zone2, test-zone1} 2021-03-02T12:24:15.252 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:15.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:15.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:15.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:15.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:15.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:15.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:15.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:15.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:15.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:15.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:15.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:15.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:15.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:15.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:15.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:15.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:15.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:15.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:15.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:15.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:15.563 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:15.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:15.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:15.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:15.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:16.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:16.119 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:16.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:16.290 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:16.398 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:16.451 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:16.562 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:16.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:16.722 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:16.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:16.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:17.007 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:17.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:17.232 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:17.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:17.408 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:17.520 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-59 zones={test-zone2, test-zone1} 2021-03-02T12:24:17.520 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-59 zones={test-zone3, test-zone1} 2021-03-02T12:24:17.530 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:17.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:17.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:17.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:17.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:17.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:17.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:17.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:17.541 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:17.541 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:17.548 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:17.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:17.549 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:17.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:17.550 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:17.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:17.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:17.551 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:17.568 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:17.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:17.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:17.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:17.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:18.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:18.179 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:18.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:18.355 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:18.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:18.527 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:18.684 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:18.817 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:19.017 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:19.130 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:19.241 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:19.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:19.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:19.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:19.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:19.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:19.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:19.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:20.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:20.229 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-59 zones={test-zone3, test-zone1} 2021-03-02T12:24:20.230 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-59 zones={test-zone3, test-zone2} 2021-03-02T12:24:20.243 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:20.258 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:20.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:20.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:20.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:20.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:20.261 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:20.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:20.262 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:20.263 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:20.271 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:20.272 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:20.273 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:20.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:20.274 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:20.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:20.275 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:20.276 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:20.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:20.363 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:20.505 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:20.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:20.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:20.838 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:20.963 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:21.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:21.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:21.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:21.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:21.443 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:21.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:21.603 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:21.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:21.821 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:21.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:21.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:22.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:22.162 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:22.274 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:22.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:22.442 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:22.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:22.669 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-59 zones={test-zone3, test-zone2} 2021-03-02T12:24:22.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:22.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:22.671 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:22.713 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:22.958 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.76.5"},{"key":1,"val":"00000000004.209.1"},{"key":2,"val":"00000000006.63.5"},{"key":3,"val":"00000000004.79.1"},{"key":4,"val":"00000000006.48.5"},{"key":5,"val":"00000000004.59.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.41.5"},{"key":8,"val":"00000000004.39.1"},{"key":9,"val":"00000000004.32.1"},{"key":10,"val":"00000000004.45.1"}]} 2021-03-02T12:24:22.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:24:22.960 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:22.961 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:22.961 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:23.011 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:23.011 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:23.260 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.76.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.209.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.63.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.32.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}}] 2021-03-02T12:24:23.261 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:24:23.261 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:23.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:23.262 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:23.306 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:23.306 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:23.540 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.76.5"},{"key":1,"val":"00000000004.209.1"},{"key":2,"val":"00000000006.63.5"},{"key":3,"val":"00000000004.79.1"},{"key":4,"val":"00000000006.48.5"},{"key":5,"val":"00000000004.59.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.41.5"},{"key":8,"val":"00000000004.39.1"},{"key":9,"val":"00000000004.32.1"},{"key":10,"val":"00000000004.45.1"}]} 2021-03-02T12:24:23.540 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:24:23.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:23.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:23.542 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:23.628 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:23.629 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:23.888 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.76.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.209.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.63.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.41.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.32.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}}] 2021-03-02T12:24:23.888 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:24:23.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:23.889 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:23.890 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:23.931 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:23.932 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:24.166 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.76.5"},{"key":1,"val":"00000000004.209.1"},{"key":2,"val":"00000000006.63.5"},{"key":3,"val":"00000000004.79.1"},{"key":4,"val":"00000000006.48.5"},{"key":5,"val":"00000000004.59.1"},{"key":6,"val":"00000000004.64.1"},{"key":7,"val":"00000000006.41.5"},{"key":8,"val":"00000000004.39.1"},{"key":9,"val":"00000000004.32.1"},{"key":10,"val":"00000000004.45.1"}]} 2021-03-02T12:24:24.166 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:24:24.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:24.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:24.168 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:24.213 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:24.214 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:24.478 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.76.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.209.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.63.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.79.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.59.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.64.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.41.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.39.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.32.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}}] 2021-03-02T12:24:24.479 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-60 2021-03-02T12:24:24.480 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:24.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:24.482 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:24.559 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:24.559 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:24.780 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000002.82.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.35.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:24.781 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:24:24.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:24.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:24.782 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:24.829 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:24.829 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:25.069 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.82.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.35.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:25.069 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:24:25.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.071 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:25.113 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:25.114 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:25.335 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000002.82.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.49.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.35.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:25.336 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:24:25.336 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.337 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:25.384 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:25.385 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:25.632 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.82.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.35.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:25.633 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-60 2021-03-02T12:24:25.633 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.634 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.634 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:25.675 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:25.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000002.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.34.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:25.919 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:24:25.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:25.920 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:26.005 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:26.006 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:26.259 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:26.259 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:24:26.260 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:26.261 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-60', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:26.262 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-60 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:26.345 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:26.346 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:26.562 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000002.68.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.34.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.34.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:26.562 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:24:26.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:26.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-60', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:26.564 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-60 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:26.608 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:26.609 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:26.838 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.68.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.34.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:26.839 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-60 2021-03-02T12:24:26.839 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-60 zones={test-zone2, test-zone1} 2021-03-02T12:24:26.850 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:26.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:26.857 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:26.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:26.858 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:26.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:26.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:26.859 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:26.860 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:26.860 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:26.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:26.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:26.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:26.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:26.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:26.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:26.873 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:26.874 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:26.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:26.998 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:27.110 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:27.220 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:27.346 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:27.470 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:27.628 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:27.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:27.806 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:27.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:27.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:28.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:28.277 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:28.331 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:28.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:28.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:28.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:28.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:28.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:28.895 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:29.008 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:29.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:29.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:29.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:29.456 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-60 zones={test-zone2, test-zone1} 2021-03-02T12:24:29.457 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-60 zones={test-zone3, test-zone1} 2021-03-02T12:24:29.466 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:29.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:29.473 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:29.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:29.474 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:29.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:29.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:29.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:29.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:29.476 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:29.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:29.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:29.483 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:29.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:29.484 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:29.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:29.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:29.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:29.501 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:29.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:29.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:29.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:29.840 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:30.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:30.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:30.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:30.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:30.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:30.618 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:30.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:30.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:30.957 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:31.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:31.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:31.298 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:31.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:31.557 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:31.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:31.753 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:31.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:31.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:32.031 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:32.139 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-60 zones={test-zone3, test-zone1} 2021-03-02T12:24:32.140 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-60 zones={test-zone3, test-zone2} 2021-03-02T12:24:32.152 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:32.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:32.159 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:32.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:32.160 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:32.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:32.161 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:32.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:32.162 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:32.162 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:32.169 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:32.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:32.170 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:32.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:32.171 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:32.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:32.172 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:32.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:32.186 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:32.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:32.362 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:32.469 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:32.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:32.635 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:32.743 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:32.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:32.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:33.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:33.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:33.184 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:33.293 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:33.347 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:33.459 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:33.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:33.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:33.733 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:33.849 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:33.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:34.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:34.137 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:34.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:34.307 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:34.425 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-60 zones={test-zone3, test-zone2} 2021-03-02T12:24:34.426 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:34.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:34.427 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:34.471 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:34.471 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:34.709 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.213.1"},{"key":2,"val":"00000000006.85.5"},{"key":3,"val":"00000000004.61.1"},{"key":4,"val":"00000000006.70.5"},{"key":5,"val":"00000000004.72.1"},{"key":6,"val":"00000000004.62.1"},{"key":7,"val":"00000000006.56.5"},{"key":8,"val":"00000000004.98.1"},{"key":9,"val":"00000000004.27.1"},{"key":10,"val":"00000000004.57.1"}]} 2021-03-02T12:24:34.710 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:24:34.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:34.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:34.712 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:34.760 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:34.761 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:35.018 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.213.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.85.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.27.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}}] 2021-03-02T12:24:35.019 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:24:35.019 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.020 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.020 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:35.061 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:35.062 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:35.298 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.213.1"},{"key":2,"val":"00000000006.85.5"},{"key":3,"val":"00000000004.61.1"},{"key":4,"val":"00000000006.70.5"},{"key":5,"val":"00000000004.72.1"},{"key":6,"val":"00000000004.62.1"},{"key":7,"val":"00000000006.56.5"},{"key":8,"val":"00000000004.98.1"},{"key":9,"val":"00000000004.27.1"},{"key":10,"val":"00000000004.57.1"}]} 2021-03-02T12:24:35.299 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:24:35.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.301 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.301 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:35.387 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:35.388 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:35.629 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.213.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.85.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.70.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.27.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}}] 2021-03-02T12:24:35.630 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:24:35.631 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.631 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.632 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:35.673 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:35.908 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.48.5"},{"key":1,"val":"00000000004.213.1"},{"key":2,"val":"00000000006.85.5"},{"key":3,"val":"00000000004.61.1"},{"key":4,"val":"00000000006.70.5"},{"key":5,"val":"00000000004.72.1"},{"key":6,"val":"00000000004.62.1"},{"key":7,"val":"00000000006.56.5"},{"key":8,"val":"00000000004.98.1"},{"key":9,"val":"00000000004.27.1"},{"key":10,"val":"00000000004.57.1"}]} 2021-03-02T12:24:35.908 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:24:35.909 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:35.910 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:35.954 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:35.954 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:36.220 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.213.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.85.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.70.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.72.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.27.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}}] 2021-03-02T12:24:36.221 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-61 2021-03-02T12:24:36.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:36.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:36.222 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:36.305 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:36.305 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:36.521 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000002.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.82.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.73.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:36.522 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:24:36.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:36.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:36.523 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:36.569 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:36.570 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:36.808 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.82.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:36.808 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:24:36.813 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:36.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:36.814 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:36.855 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:36.855 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:37.073 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000002.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.82.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.73.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:37.073 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:24:37.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:37.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:37.075 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:37.121 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:37.122 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:37.367 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.82.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.73.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:37.447 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-61 2021-03-02T12:24:37.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:37.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:37.449 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:37.494 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:37.495 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:37.723 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000002.41.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.84.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.76.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:37.724 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:24:37.724 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:37.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:37.725 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:37.813 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:38.065 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.84.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:38.066 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:24:38.066 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:38.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:38.071 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:38.118 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:38.118 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:38.344 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000002.41.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.84.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.76.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.51.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:38.344 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:24:38.345 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:38.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:38.346 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:38.391 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:38.392 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:38.630 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.41.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.84.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.76.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.51.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:38.764 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-61 2021-03-02T12:24:38.764 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-61 zones={test-zone2, test-zone1} 2021-03-02T12:24:38.776 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:38.782 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:38.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:38.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:38.783 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:38.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:38.784 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:38.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:38.785 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:38.785 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:38.794 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:38.795 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:38.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:38.796 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:38.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:38.797 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:38.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:38.798 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:38.811 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:38.863 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:38.977 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:39.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:39.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:39.335 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:39.502 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:39.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:39.668 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:39.779 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:39.834 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:39.954 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:40.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:40.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:40.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:40.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:40.485 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:40.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:40.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:40.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:41.010 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:41.128 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:41.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:41.344 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:41.466 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-61 zones={test-zone2, test-zone1} 2021-03-02T12:24:41.467 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-61 zones={test-zone3, test-zone1} 2021-03-02T12:24:41.479 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:41.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:41.495 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:41.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:41.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:41.496 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:41.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:41.497 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:41.498 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:41.498 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:41.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:41.507 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:41.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:41.508 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:41.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:41.509 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:41.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:41.510 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:41.531 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:41.601 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:41.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:41.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:41.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:42.076 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:42.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:42.266 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:42.397 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:42.518 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:42.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:42.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:42.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:42.871 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:42.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:43.096 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:43.151 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:43.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:43.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:43.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:43.553 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:43.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:43.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:43.846 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:43.951 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-61 zones={test-zone3, test-zone1} 2021-03-02T12:24:43.952 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-61 zones={test-zone3, test-zone2} 2021-03-02T12:24:43.961 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:43.969 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:43.970 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:43.971 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:43.972 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:43.972 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:43.979 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:43.981 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:43.982 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:43.983 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:24:43.984 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:24:43.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:24:43.985 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:24:44.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:44.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:44.166 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:44.278 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:44.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:44.456 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:44.574 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:44.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:44.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:44.847 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:44.907 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:45.018 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:45.122 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:45.175 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:45.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:24:45.409 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:45.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:45.579 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:24:45.685 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:45.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:45.854 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:24:45.961 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:46.020 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:46.135 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:24:46.250 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-61 zones={test-zone3, test-zone2} 2021-03-02T12:24:46.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:46.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:46.253 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:46.294 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:46.295 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:46.523 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687846.845478_411.1","last_update":"2021-03-02T12:24:06.845478Z"},{"marker":"1_1614687845.199691_618.3","last_update":"2021-03-02T12:24:05.199691Z"},{"marker":"1_1614687846.806394_396.1","last_update":"2021-03-02T12:24:06.806394Z"},{"marker":"1_1614687847.812738_496.1","last_update":"2021-03-02T12:24:07.812738Z"}] 2021-03-02T12:24:46.523 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:24:46.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:46.524 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:46.525 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:46.569 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:46.570 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:46.793 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.845478_411.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.845478Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687845.199691_618.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:05.199691Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.806394_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.806394Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687847.812738_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:07.812738Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:46.793 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:24:46.794 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:46.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:46.796 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:46.836 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:47.078 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687846.845478_411.1","last_update":"2021-03-02T12:24:06.845478Z"},{"marker":"1_1614687845.199691_618.3","last_update":"2021-03-02T12:24:05.199691Z"},{"marker":"1_1614687846.806394_396.1","last_update":"2021-03-02T12:24:06.806394Z"},{"marker":"1_1614687847.812738_496.1","last_update":"2021-03-02T12:24:07.812738Z"}] 2021-03-02T12:24:47.079 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:24:47.079 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:47.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:47.080 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:47.162 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:47.163 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:47.400 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7196637147885404157},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.845478_411.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.845478Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687845.199691_618.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:05.199691Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.806394_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.806394Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687847.812738_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:07.812738Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:47.401 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:24:47.402 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:47.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:47.403 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:47.486 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:47.486 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:47.724 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687846.845478_411.1","last_update":"2021-03-02T12:24:06.845478Z"},{"marker":"1_1614687845.199691_618.3","last_update":"2021-03-02T12:24:05.199691Z"},{"marker":"1_1614687846.806394_396.1","last_update":"2021-03-02T12:24:06.806394Z"},{"marker":"1_1614687847.812738_496.1","last_update":"2021-03-02T12:24:07.812738Z"}] 2021-03-02T12:24:47.724 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:24:47.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:47.729 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:47.730 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:47.771 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:47.772 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:48.040 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.845478_411.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.845478Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687845.199691_618.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:05.199691Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.806394_396.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.806394Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687847.812738_496.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:07.812738Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:48.041 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:24:48.042 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.042 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.043 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:48.122 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:48.122 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:48.336 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687846.883573_304.1","last_update":"2021-03-02T12:24:06.883573Z"},{"marker":"1_1614687709.020343_566.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687846.886527_432.1","last_update":"2021-03-02T12:24:06.886527Z"},{"marker":"1_1614687847.812741_380.1","last_update":"2021-03-02T12:24:07.812741Z"}] 2021-03-02T12:24:48.336 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:24:48.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.338 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:48.381 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:48.382 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:48.618 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.883573_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.883573Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_566.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.886527_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.886527Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687847.812741_380.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:07.812741Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:48.619 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:24:48.619 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.620 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.620 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:48.662 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:48.662 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:48.867 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687846.883573_304.1","last_update":"2021-03-02T12:24:06.883573Z"},{"marker":"1_1614687709.020343_566.3","last_update":"2021-03-02T12:21:49.020343Z"},{"marker":"1_1614687846.886527_432.1","last_update":"2021-03-02T12:24:06.886527Z"},{"marker":"1_1614687847.812741_380.1","last_update":"2021-03-02T12:24:07.812741Z"}] 2021-03-02T12:24:48.867 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:24:48.868 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:48.869 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:48.915 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:48.916 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:49.157 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13674358676509122085},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.883573_304.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.883573Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687709.020343_566.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:21:49.020343Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.886527_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.886527Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687847.812741_380.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:07.812741Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:49.157 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:24:49.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:49.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:49.159 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:49.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:49.236 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:49.466 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687846.560993_191.1","last_update":"2021-03-02T12:24:06.560993Z"},{"marker":"1_1614687751.312219_502.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687846.587411_394.1","last_update":"2021-03-02T12:24:06.587411Z"},{"marker":"1_1614687846.827879_265.1","last_update":"2021-03-02T12:24:06.827879Z"}] 2021-03-02T12:24:49.560 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:24:49.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:49.561 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:49.562 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:49.605 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:49.606 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:49.845 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14630138824005222133},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.560993_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.560993Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_502.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.587411_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.587411Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687846.827879_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.827879Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:49.942 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:24:49.943 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:49.943 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:49.944 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:49.988 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:49.988 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:50.213 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687846.560993_191.1","last_update":"2021-03-02T12:24:06.560993Z"},{"marker":"1_1614687751.312219_502.3","last_update":"2021-03-02T12:22:31.312219Z"},{"marker":"1_1614687846.587411_394.1","last_update":"2021-03-02T12:24:06.587411Z"},{"marker":"1_1614687846.827879_265.1","last_update":"2021-03-02T12:24:06.827879Z"}] 2021-03-02T12:24:50.246 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:24:50.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:50.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:50.248 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:50.292 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:50.292 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:50.509 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9850944892582908325},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687846.560993_191.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.560993Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687751.312219_502.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:22:31.312219Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687846.587411_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.587411Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687846.827879_265.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:06.827879Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:24:50.510 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:24:50.511 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-03-02T12:24:50.513 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-62 2021-03-02T12:24:50.554 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-63 2021-03-02T12:24:50.622 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-64 2021-03-02T12:24:51.823 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:51.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:51.827 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:51.869 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:52.080 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:24:52.116 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614687891.020237_527.1","last_update":"2021-03-02T12:24:51.020237Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:24:52.117 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:24:52.117 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.118 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.118 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:52.165 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:52.166 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:52.343 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687891.020237_527.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:51.020237Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:24:52.344 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:24:52.344 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:24:52.345 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.346 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:52.389 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:52.389 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:52.656 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687891.020237_527.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:51.020237Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:24:52.656 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:24:52.657 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:24:52.657 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.658 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:52.704 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:52.705 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:52.961 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687891.020237_527.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:51.020237Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:24:52.961 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:24:52.962 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:52.963 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:53.008 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:53.009 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:53.234 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:53.235 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:24:53.236 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:53.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:53.237 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:53.282 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:53.282 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:53.513 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:53.514 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:24:53.514 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:53.515 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:53.516 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:53.557 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:53.557 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:53.751 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:53.751 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:24:53.752 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:53.752 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:53.753 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:53.832 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:54.053 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:54.054 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:24:54.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.055 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:54.134 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:54.350 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.36.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.67.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.60.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.45.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:54.351 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:24:54.352 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.353 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:54.397 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:54.397 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:54.663 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.36.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:54.663 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:24:54.664 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.664 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.665 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:54.741 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:54.741 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:54.958 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.57.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:54.959 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:24:54.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.960 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:54.960 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:55.006 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:55.242 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:55.243 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:24:55.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:55.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:55.244 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:55.287 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:55.288 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:55.495 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.34.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.59.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.57.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:55.495 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:24:55.496 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:55.496 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:55.497 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:55.540 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:55.540 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:55.769 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.34.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.59.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.57.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:55.770 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:24:55.770 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:55.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:55.771 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:55.852 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:55.853 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:56.083 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:56.084 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:24:56.085 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:56.085 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:56.086 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:56.169 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:56.424 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:56.424 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:24:56.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:56.425 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:56.426 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:56.510 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:56.732 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.33.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.58.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.75.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.43.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:24:56.733 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:24:56.734 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:56.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:24:56.735 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:24:56.779 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:24:56.780 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:24:57.011 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.33.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.75.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.43.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:24:57.012 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:24:57.012 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-62 zones={test-zone2, test-zone1} 2021-03-02T12:24:57.023 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:57.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:57.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:57.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:57.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:57.030 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:57.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:57.037 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:57.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:57.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:57.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:57.102 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:57.214 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:57.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:57.384 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:57.506 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:57.611 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:57.667 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:57.780 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:57.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:57.940 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:58.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:58.159 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-62 zones={test-zone2, test-zone1} 2021-03-02T12:24:58.159 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-62 zones={test-zone3, test-zone1} 2021-03-02T12:24:58.169 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:58.175 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:58.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:58.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:58.177 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:58.177 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:58.183 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:58.184 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:58.185 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:58.198 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:58.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:58.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:58.480 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:58.536 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:58.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:58.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:58.814 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:58.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:24:59.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:59.211 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:59.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:24:59.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-62 zones={test-zone3, test-zone1} 2021-03-02T12:24:59.444 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-62 zones={test-zone3, test-zone2} 2021-03-02T12:24:59.454 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:24:59.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:59.460 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:59.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:59.461 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:59.462 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:24:59.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:24:59.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:24:59.469 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:24:59.470 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:24:59.481 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:59.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:59.747 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:24:59.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:24:59.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:00.093 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:00.251 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:00.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:00.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:00.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:00.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:00.784 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:00.906 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-62 zones={test-zone3, test-zone2} 2021-03-02T12:25:01.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:01.267 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:01.268 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:01.310 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:01.537 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.102.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:01.538 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:25:01.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:01.539 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:01.540 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:01.588 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:01.588 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:01.807 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.102.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:01.808 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:25:01.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:01.809 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:01.809 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:01.865 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:01.866 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:02.097 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.102.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:02.098 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:25:02.100 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:02.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:02.101 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:02.194 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:02.195 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:02.434 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.102.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:02.435 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:25:02.436 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:02.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:02.438 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:02.521 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:02.522 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:02.747 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.38.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.102.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:02.748 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:25:02.749 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:02.749 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:02.750 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:02.797 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:02.798 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:03.066 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.102.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:03.067 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:25:03.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.069 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:03.161 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:03.162 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:03.383 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.30.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.41.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:03.384 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:25:03.385 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.386 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.387 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:03.434 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:03.435 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:03.680 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:03.681 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:25:03.681 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.682 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:03.727 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:03.727 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:03.947 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.30.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.46.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.41.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:03.947 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:25:03.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:03.949 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:03.992 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:03.993 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:04.233 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.46.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:04.233 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:25:04.234 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:04.234 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:04.235 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:04.316 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:04.317 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:04.542 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.35.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:04.542 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:25:04.543 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:04.543 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:04.544 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:04.626 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:04.627 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:04.870 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:04.871 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:25:04.872 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:04.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:04.873 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:04.955 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:04.955 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:05.177 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.35.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.37.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.67.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:05.178 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:25:05.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:05.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:05.181 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:05.227 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:05.227 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:05.457 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.67.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:05.458 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:25:05.458 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-63 zones={test-zone2, test-zone1} 2021-03-02T12:25:05.469 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:25:05.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:05.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:05.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:05.477 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:05.478 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:25:05.485 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:05.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:05.486 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:05.487 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:05.500 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:05.554 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:05.708 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:05.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:05.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:05.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:06.107 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:06.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:06.276 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:06.389 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:06.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:06.564 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:06.667 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-63 zones={test-zone2, test-zone1} 2021-03-02T12:25:06.667 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-63 zones={test-zone3, test-zone1} 2021-03-02T12:25:06.676 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:25:06.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:06.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:06.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:06.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:06.684 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:25:06.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:06.689 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:06.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:06.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:06.702 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:06.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:06.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:06.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:07.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:07.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:07.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:07.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:07.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:07.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:07.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:07.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:07.822 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-63 zones={test-zone3, test-zone1} 2021-03-02T12:25:07.822 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-63 zones={test-zone3, test-zone2} 2021-03-02T12:25:07.833 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:25:07.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:07.883 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:07.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:07.884 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:07.885 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:25:07.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:07.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:07.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:07.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:07.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:07.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:08.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:08.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:08.239 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:08.348 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:08.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:08.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:08.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:08.729 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:08.783 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:08.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:09.009 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-63 zones={test-zone3, test-zone2} 2021-03-02T12:25:09.009 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.010 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:09.053 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:09.281 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:09.282 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:25:09.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.283 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:09.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:09.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:09.564 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:09.565 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:25:09.565 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.567 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:09.611 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:09.612 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:09.841 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:09.842 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:25:09.842 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.843 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:09.843 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:09.888 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:09.889 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:10.137 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:10.137 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:25:10.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:10.138 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:10.139 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:10.222 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:10.223 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:10.460 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.90.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.51.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.74.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:10.461 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:25:10.461 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:10.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:10.462 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:10.507 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:10.508 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:10.773 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.90.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.51.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.74.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:10.773 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:25:10.774 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:10.775 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:10.775 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:10.857 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:10.857 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:11.071 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.88.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:11.099 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:25:11.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:11.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:11.101 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:11.145 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:11.146 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:11.397 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.88.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:11.418 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:25:11.419 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:11.419 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:11.420 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:11.463 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:11.464 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:11.985 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000001.88.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.38.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.78.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.38.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:11.986 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:25:11.986 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:11.987 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:11.987 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:12.033 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:12.034 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:12.279 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.88.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.78.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:12.291 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:25:12.292 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:12.293 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:12.293 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:12.362 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:12.362 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:12.589 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.95.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.40.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.35.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:12.590 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:25:12.591 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:12.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:12.592 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:12.689 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:12.690 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:12.943 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:12.943 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:25:12.944 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:12.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:12.945 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:13.041 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:13.042 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:13.271 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000001.95.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.40.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.85.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.35.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:25:13.272 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:25:13.273 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:13.273 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:13.274 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:13.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:13.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:13.556 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.95.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.85.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:25:13.557 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:25:13.557 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-64 zones={test-zone2, test-zone1} 2021-03-02T12:25:13.574 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:25:13.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:13.583 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:13.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:13.584 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:13.584 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:25:13.596 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:13.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:13.597 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:13.598 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:13.622 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:13.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:13.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:13.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:14.009 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:14.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:14.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:14.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:14.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:14.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:14.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:14.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:14.847 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-64 zones={test-zone2, test-zone1} 2021-03-02T12:25:14.848 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-64 zones={test-zone3, test-zone1} 2021-03-02T12:25:14.855 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:25:14.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:14.861 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:14.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:14.862 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:14.863 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:25:14.868 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:14.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:14.869 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:14.870 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:14.881 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:14.934 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:15.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:15.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:15.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:15.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:15.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:15.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:15.633 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:15.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:15.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:15.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:16.019 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-64 zones={test-zone3, test-zone1} 2021-03-02T12:25:16.020 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-64 zones={test-zone3, test-zone2} 2021-03-02T12:25:16.031 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:25:16.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:16.038 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:16.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:16.039 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:16.039 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:25:16.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:25:16.046 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:25:16.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:25:16.047 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:25:16.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:16.114 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:16.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:25:16.377 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:16.429 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:16.542 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:25:16.646 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:16.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:16.810 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:25:16.913 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:16.971 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:17.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:25:17.193 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-64 zones={test-zone3, test-zone2} 2021-03-02T12:25:17.193 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:17.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:17.194 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:25:17.238 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:17.239 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:17.522 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:17.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:17.523 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:25:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:17.607 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:17.889 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'majrnq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:17.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'majrnq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:17.890 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket majrnq-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:25:17.973 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:17.973 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:18.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:18.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:18.264 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:18.349 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:18.349 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:18.556 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:25:18.589 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614687918.218854_565.1","last_update":"2021-03-02T12:25:18.218854Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:25:18.590 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:25:18.590 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:18.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:18.591 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:18.634 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:18.634 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:18.855 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687891.020237_527.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:51.020237Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:25:18.855 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614687918.218854_565.1 target=1_1614687891.020237_527.1 2021-03-02T12:25:23.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:23.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:23.874 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:23.922 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:23.923 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:24.142 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687918.218854_565.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:18.218854Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:25:24.142 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:25:24.143 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:25:24.143 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:24.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:24.144 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:24.189 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:24.189 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:24.414 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687891.020237_527.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:51.020237Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:25:24.416 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614687918.218854_565.1 target=1_1614687891.020237_527.1 2021-03-02T12:25:29.418 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:29.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:29.440 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:29.483 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:29.483 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:29.691 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687918.218854_565.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:18.218854Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:25:29.691 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:25:29.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:25:29.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:29.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:29.693 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:29.735 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:29.736 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:30.006 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687891.020237_527.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:24:51.020237Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:25:30.006 WARNING:rgw_multi.tests:zone test-zone4 behind master: shard=1 master=1_1614687918.218854_565.1 target=1_1614687891.020237_527.1 2021-03-02T12:25:35.008 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:35.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:35.023 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:35.071 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:35.071 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:35.335 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614687918.218854_565.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:18.218854Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:25:35.336 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:25:36.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'majrnq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:36.003 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'majrnq-62', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:36.004 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket majrnq-62 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:25:36.047 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:36.047 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:36.333 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'majrnq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:36.333 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'majrnq-63', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:36.334 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket majrnq-63 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:25:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:36.430 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:36.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'majrnq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:36.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'majrnq-64', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:25:36.677 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket majrnq-64 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:25:36.719 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:36.720 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:36.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:36.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:36.993 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:37.076 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:37.076 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:37.299 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.46.1"},{"key":1,"val":"00000000004.55.1"},{"key":2,"val":"00000000006.77.1"},{"key":3,"val":"00000000004.76.1"},{"key":4,"val":"00000000006.66.1"},{"key":5,"val":"00000000004.56.1"},{"key":6,"val":"00000000004.61.1"},{"key":7,"val":"00000000006.53.1"},{"key":8,"val":"00000000004.68.1"},{"key":9,"val":"00000000004.57.1"},{"key":10,"val":"00000000004.77.1"}]} 2021-03-02T12:25:37.299 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:25:37.300 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:37.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:37.301 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:37.345 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:37.346 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:37.588 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"full-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:25:37.589 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.55.1 target=, shard=5 master=00000000004.56.1 target=, shard=9 master=00000000004.57.1 target= 2021-03-02T12:25:42.591 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:42.591 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:42.592 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:42.643 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:42.644 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:42.842 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:25:43.419 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.55.1 target=, shard=5 master=00000000004.56.1 target=, shard=9 master=00000000004.57.1 target= 2021-03-02T12:25:48.421 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:48.433 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:48.433 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:48.478 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:48.479 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:48.719 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:25:48.720 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:25:48.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:48.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:48.721 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:48.765 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:49.030 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.46.1"},{"key":1,"val":"00000000004.55.1"},{"key":2,"val":"00000000006.77.1"},{"key":3,"val":"00000000004.76.1"},{"key":4,"val":"00000000006.66.1"},{"key":5,"val":"00000000004.56.1"},{"key":6,"val":"00000000004.61.1"},{"key":7,"val":"00000000006.53.1"},{"key":8,"val":"00000000004.68.1"},{"key":9,"val":"00000000004.57.1"},{"key":10,"val":"00000000004.77.1"}]} 2021-03-02T12:25:49.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:25:49.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:49.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:49.032 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:49.116 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:49.117 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:49.372 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.46.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.66.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:25:49.372 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:25:49.373 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:49.373 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:49.374 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:49.455 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:49.456 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:49.700 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.46.1"},{"key":1,"val":"00000000004.55.1"},{"key":2,"val":"00000000006.77.1"},{"key":3,"val":"00000000004.76.1"},{"key":4,"val":"00000000006.66.1"},{"key":5,"val":"00000000004.56.1"},{"key":6,"val":"00000000004.61.1"},{"key":7,"val":"00000000006.53.1"},{"key":8,"val":"00000000004.68.1"},{"key":9,"val":"00000000004.57.1"},{"key":10,"val":"00000000004.77.1"}]} 2021-03-02T12:25:49.701 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:25:49.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:49.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:49.703 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:49.748 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:49.749 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:50.011 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:25:50.012 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:25:55.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:55.021 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:25:55.022 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:25:55.069 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:25:55.070 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:25:55.338 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:25:55.383 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:00.385 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:00.396 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:00.397 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:00.442 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:00.443 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:00.709 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:00.710 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:05.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:05.726 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:05.727 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:05.773 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:05.773 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:06.038 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:06.039 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:11.040 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:11.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:11.048 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:11.092 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:11.093 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:11.360 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:11.361 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:16.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:16.364 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:16.365 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:16.412 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:16.412 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:16.674 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:16.674 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:21.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:21.688 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:21.689 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:21.738 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:21.738 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:22.003 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:22.003 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:27.005 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:27.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:27.020 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:27.067 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:27.067 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:27.338 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:27.339 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:32.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:32.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:32.353 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:32.406 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:32.406 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:32.673 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:32.851 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:37.852 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:37.853 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:37.854 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:37.899 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:37.900 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:38.164 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:38.165 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:43.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:43.172 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:43.173 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:43.220 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:43.221 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:43.487 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:43.488 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:48.491 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:48.491 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:48.492 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:48.538 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:48.538 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:48.808 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:48.809 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:53.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:53.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:53.813 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:53.864 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:53.865 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:54.124 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:54.266 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:26:59.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:59.269 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:26:59.270 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:26:59.315 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:26:59.316 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:26:59.580 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:26:59.581 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:04.582 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:04.583 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:04.584 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:04.629 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:04.630 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:04.893 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:04.975 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:09.976 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:10.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:10.064 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:10.109 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:10.110 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:10.377 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:10.378 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:15.380 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:15.381 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:15.382 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:15.429 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:15.430 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:15.698 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:15.699 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:20.700 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:20.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:20.716 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:20.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:20.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:21.044 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:21.045 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:26.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:26.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:26.062 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:26.111 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:26.112 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:26.382 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:26.383 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:31.384 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:31.385 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:31.386 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:31.447 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:31.448 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:31.714 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:31.715 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:36.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:36.732 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:36.733 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:36.778 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:36.779 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:37.043 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:37.044 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:42.045 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:42.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:42.054 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:42.111 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:42.111 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:42.376 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:42.376 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:47.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:47.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:47.383 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:47.428 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:47.429 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:47.694 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:47.695 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=2 master=00000000006.77.1 target=00000000002.68.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:52.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:52.697 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:52.698 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:52.763 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:52.763 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:53.029 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:53.030 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=4 master=00000000006.66.1 target=00000000002.61.1, shard=7 master=00000000006.53.1 target=00000000002.46.1 2021-03-02T12:27:58.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:58.145 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:27:58.146 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:27:58.196 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:27:58.197 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:27:58.468 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:27:58.635 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=4 master=00000000006.66.1 target=00000000002.61.1 2021-03-02T12:28:03.637 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:03.648 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:03.649 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:03.698 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:03.698 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:03.971 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:28:03.971 WARNING:rgw_multi.tests:bucket majrnq-62 zone test-zone4 behind zone test-zone1: shard=0 master=00000000006.46.1 target=00000000002.39.1, shard=4 master=00000000006.66.1 target=00000000002.61.1 2021-03-02T12:28:08.973 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:09.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:09.032 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:09.078 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:09.078 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:09.447 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.77.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.76.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.66.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.61.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.53.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.68.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.57.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.77.1"}}}] 2021-03-02T12:28:09.447 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-62 2021-03-02T12:28:09.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:09.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:09.449 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:09.531 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:09.532 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:09.757 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000004.44.1"},{"key":1,"val":"00000000002.39.1"},{"key":2,"val":"00000000004.70.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.62.1"},{"key":5,"val":"00000000002.42.1"},{"key":6,"val":"00000000002.49.1"},{"key":7,"val":"00000000004.50.1"},{"key":8,"val":"00000000002.54.1"},{"key":9,"val":"00000000002.52.1"},{"key":10,"val":"00000000002.83.1"}]} 2021-03-02T12:28:09.807 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:28:09.808 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:09.808 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:09.809 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:09.855 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:09.855 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:10.133 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.44.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.62.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.50.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}}] 2021-03-02T12:28:10.134 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:28:10.135 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:10.135 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:10.136 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:10.178 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:10.178 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:10.407 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000004.44.1"},{"key":1,"val":"00000000002.39.1"},{"key":2,"val":"00000000004.70.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.62.1"},{"key":5,"val":"00000000002.42.1"},{"key":6,"val":"00000000002.49.1"},{"key":7,"val":"00000000004.50.1"},{"key":8,"val":"00000000002.54.1"},{"key":9,"val":"00000000002.52.1"},{"key":10,"val":"00000000002.83.1"}]} 2021-03-02T12:28:10.407 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:28:10.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:10.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:10.409 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:10.458 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:10.459 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:10.701 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.44.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.39.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.70.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.62.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.50.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}}] 2021-03-02T12:28:10.783 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-62 2021-03-02T12:28:10.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:10.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:10.784 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:10.827 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:11.059 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000004.45.1"},{"key":1,"val":"00000000002.43.1"},{"key":2,"val":"00000000004.69.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.80.1"},{"key":5,"val":"00000000002.46.1"},{"key":6,"val":"00000000002.45.1"},{"key":7,"val":"00000000004.51.1"},{"key":8,"val":"00000000002.55.1"},{"key":9,"val":"00000000002.55.1"},{"key":10,"val":"00000000002.83.1"}]} 2021-03-02T12:28:11.075 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:28:11.075 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:11.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:11.077 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:11.145 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:11.145 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:11.412 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.45.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.69.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}}] 2021-03-02T12:28:11.413 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:28:11.414 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:11.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:11.415 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:11.498 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:11.721 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000004.45.1"},{"key":1,"val":"00000000002.43.1"},{"key":2,"val":"00000000004.69.1"},{"key":3,"val":"00000000002.80.1"},{"key":4,"val":"00000000004.80.1"},{"key":5,"val":"00000000002.46.1"},{"key":6,"val":"00000000002.45.1"},{"key":7,"val":"00000000004.51.1"},{"key":8,"val":"00000000002.55.1"},{"key":9,"val":"00000000002.55.1"},{"key":10,"val":"00000000002.83.1"}]} 2021-03-02T12:28:11.722 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:28:11.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:11.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:11.723 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:11.768 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:11.769 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:12.006 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.45.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.43.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.69.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.80.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.55.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}}] 2021-03-02T12:28:12.007 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-62 2021-03-02T12:28:12.007 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-62 zones={test-zone2, test-zone1} 2021-03-02T12:28:12.021 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:12.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:12.028 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:12.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:12.029 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:12.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:12.030 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:12.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:12.031 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:12.032 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:12.041 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:12.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:12.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:12.042 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:12.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:12.043 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:12.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:12.044 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:12.057 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:12.118 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:12.260 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:12.368 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:12.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:12.561 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:12.677 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:12.725 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:12.836 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:12.946 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:13.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:13.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:13.435 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:13.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:13.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:13.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:13.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:14.080 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:14.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:14.280 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:14.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:14.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:14.615 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:14.759 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:14.889 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-62 zones={test-zone2, test-zone1} 2021-03-02T12:28:14.889 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-62 zones={test-zone3, test-zone1} 2021-03-02T12:28:14.911 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:14.923 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:14.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:14.925 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:14.926 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:14.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:14.928 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:14.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:14.929 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:14.930 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:14.945 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:14.946 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:14.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:14.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:14.949 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:14.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:14.950 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:14.951 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:14.982 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:15.052 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:15.196 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:15.312 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:15.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:15.490 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:15.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:15.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:15.785 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:15.892 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:15.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:16.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:16.170 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:16.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:16.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:16.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:16.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:16.605 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:16.716 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:16.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:16.887 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:17.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:17.055 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:17.169 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:17.274 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-62 zones={test-zone3, test-zone1} 2021-03-02T12:28:17.275 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-62 zones={test-zone3, test-zone2} 2021-03-02T12:28:17.285 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:17.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:17.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:17.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:17.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:17.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:17.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:17.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:17.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:17.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:17.302 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:17.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:17.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:17.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:17.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:17.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:17.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:17.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:17.319 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:17.371 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:17.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:17.589 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:17.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:17.758 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:17.869 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:17.928 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:18.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:18.149 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:18.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:18.316 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:18.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:18.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:18.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:18.701 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:18.756 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:18.866 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:18.969 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:19.028 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:19.146 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:19.253 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:19.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:19.427 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:19.545 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-62 zones={test-zone3, test-zone2} 2021-03-02T12:28:19.548 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:19.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:19.563 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:19.605 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:19.605 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:19.838 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.50.5"},{"key":1,"val":"00000000004.35.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.63.1"},{"key":4,"val":"00000000006.108.5"},{"key":5,"val":"00000000004.45.1"},{"key":6,"val":"00000000004.80.1"},{"key":7,"val":"00000000006.48.5"},{"key":8,"val":"00000000004.50.1"},{"key":9,"val":"00000000004.98.1"},{"key":10,"val":"00000000004.84.1"}]} 2021-03-02T12:28:19.838 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:28:19.839 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:19.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:19.841 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:19.885 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:19.885 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:20.126 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.50.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.108.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.48.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}}] 2021-03-02T12:28:20.126 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:28:20.127 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:20.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:20.128 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:20.171 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:20.425 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.50.5"},{"key":1,"val":"00000000004.35.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.63.1"},{"key":4,"val":"00000000006.108.5"},{"key":5,"val":"00000000004.45.1"},{"key":6,"val":"00000000004.80.1"},{"key":7,"val":"00000000006.48.5"},{"key":8,"val":"00000000004.50.1"},{"key":9,"val":"00000000004.98.1"},{"key":10,"val":"00000000004.84.1"}]} 2021-03-02T12:28:20.488 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:28:20.489 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:20.489 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:20.490 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:20.535 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:20.536 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:20.765 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.50.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.108.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.48.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}}] 2021-03-02T12:28:20.780 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:28:20.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:20.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:20.782 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:20.824 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:20.825 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:21.073 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.50.5"},{"key":1,"val":"00000000004.35.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.63.1"},{"key":4,"val":"00000000006.108.5"},{"key":5,"val":"00000000004.45.1"},{"key":6,"val":"00000000004.80.1"},{"key":7,"val":"00000000006.48.5"},{"key":8,"val":"00000000004.50.1"},{"key":9,"val":"00000000004.98.1"},{"key":10,"val":"00000000004.84.1"}]} 2021-03-02T12:28:21.079 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:28:21.079 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:21.080 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:21.080 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:21.126 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:21.127 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:21.390 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.50.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.108.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.50.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.98.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.84.1"}}}] 2021-03-02T12:28:21.405 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-63 2021-03-02T12:28:21.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:21.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:21.407 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:21.477 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:21.478 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:21.719 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000004.43.1"},{"key":1,"val":"00000000002.27.1"},{"key":2,"val":"00000000004.52.1"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.88.1"},{"key":5,"val":"00000000002.50.1"},{"key":6,"val":"00000000002.84.1"},{"key":7,"val":"00000000004.51.1"},{"key":8,"val":"00000000002.49.1"},{"key":9,"val":"00000000002.97.1"},{"key":10,"val":"00000000002.93.1"}]} 2021-03-02T12:28:21.720 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:28:21.720 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:21.721 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:21.721 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:21.767 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:22.020 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.43.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.88.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.97.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.93.1"}}}] 2021-03-02T12:28:22.021 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:28:22.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.023 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:22.065 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:22.065 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:22.293 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000004.43.1"},{"key":1,"val":"00000000002.27.1"},{"key":2,"val":"00000000004.52.1"},{"key":3,"val":"00000000002.47.1"},{"key":4,"val":"00000000004.88.1"},{"key":5,"val":"00000000002.50.1"},{"key":6,"val":"00000000002.84.1"},{"key":7,"val":"00000000004.51.1"},{"key":8,"val":"00000000002.49.1"},{"key":9,"val":"00000000002.97.1"},{"key":10,"val":"00000000002.93.1"}]} 2021-03-02T12:28:22.294 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:28:22.294 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.296 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:22.343 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:22.532 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:22.590 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.43.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.52.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.47.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.88.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.51.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.97.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.93.1"}}}] 2021-03-02T12:28:22.591 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-63 2021-03-02T12:28:22.592 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.594 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:22.672 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:22.673 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:22.918 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000004.44.1"},{"key":1,"val":"00000000002.25.1"},{"key":2,"val":"00000000004.42.1"},{"key":3,"val":"00000000002.42.1"},{"key":4,"val":"00000000004.72.1"},{"key":5,"val":"00000000002.46.1"},{"key":6,"val":"00000000002.92.1"},{"key":7,"val":"00000000004.47.1"},{"key":8,"val":"00000000002.46.1"},{"key":9,"val":"00000000002.104.1"},{"key":10,"val":"00000000002.80.1"}]} 2021-03-02T12:28:22.919 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:28:22.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:22.920 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:23.007 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:23.008 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:23.264 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.44.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.25.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.92.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.47.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.104.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}}] 2021-03-02T12:28:23.299 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:28:23.300 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:23.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:23.301 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:23.347 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:23.348 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:23.584 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000004.44.1"},{"key":1,"val":"00000000002.25.1"},{"key":2,"val":"00000000004.42.1"},{"key":3,"val":"00000000002.42.1"},{"key":4,"val":"00000000004.72.1"},{"key":5,"val":"00000000002.46.1"},{"key":6,"val":"00000000002.92.1"},{"key":7,"val":"00000000004.47.1"},{"key":8,"val":"00000000002.46.1"},{"key":9,"val":"00000000002.104.1"},{"key":10,"val":"00000000002.80.1"}]} 2021-03-02T12:28:23.585 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:28:23.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:23.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:23.587 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:23.632 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:23.633 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:23.872 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.44.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.25.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.42.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.92.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.47.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.46.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.104.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.1"}}}] 2021-03-02T12:28:23.872 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-63 2021-03-02T12:28:23.873 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-63 zones={test-zone2, test-zone1} 2021-03-02T12:28:23.884 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:23.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:23.891 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:23.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:23.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:23.892 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:23.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:23.893 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:23.894 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:23.894 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:23.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:23.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:23.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:23.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:23.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:23.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:23.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:23.905 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:23.920 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:23.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:24.235 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:24.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:24.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:24.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:24.682 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:24.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:24.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:24.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:25.063 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:25.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:25.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:25.378 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:25.537 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:25.664 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:25.740 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:25.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:26.015 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:26.073 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:26.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:26.309 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:26.364 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:26.482 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:26.592 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-63 zones={test-zone2, test-zone1} 2021-03-02T12:28:26.593 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-63 zones={test-zone3, test-zone1} 2021-03-02T12:28:26.605 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:26.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:26.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:26.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:26.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:26.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:26.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:26.616 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:26.617 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:26.617 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:26.624 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:26.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:26.625 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:26.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:26.626 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:26.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:26.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:26.627 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:26.644 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:26.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:26.816 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:26.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:26.980 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:27.097 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:27.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:27.256 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:27.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:27.479 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:27.535 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:27.645 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:27.762 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:27.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:27.936 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:28.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:28.103 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:28.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:28.322 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:28.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:28.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:28.599 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:28.656 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:28.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:28.883 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-63 zones={test-zone3, test-zone1} 2021-03-02T12:28:28.884 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-63 zones={test-zone3, test-zone2} 2021-03-02T12:28:28.894 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:28.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:28.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:28.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:28.902 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:28.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:28.903 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:28.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:28.904 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:28.905 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:28.911 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:28.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:28.912 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:28.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:28.913 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:28.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:28.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:28.914 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:28.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:28.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:29.098 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:29.200 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:29.259 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:29.375 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:29.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:29.546 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:29.660 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:29.769 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:29.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:29.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:30.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:30.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:30.218 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:30.321 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:30.376 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:30.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:30.621 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:30.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:30.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:30.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:30.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:31.082 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:31.209 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-63 zones={test-zone3, test-zone2} 2021-03-02T12:28:31.210 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:31.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:31.211 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:31.255 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:31.255 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:31.511 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.104.5"},{"key":1,"val":"00000000004.55.1"},{"key":2,"val":"00000000006.57.5"},{"key":3,"val":"00000000004.102.1"},{"key":4,"val":"00000000006.84.5"},{"key":5,"val":"00000000004.80.1"},{"key":6,"val":"00000000004.86.1"},{"key":7,"val":"00000000006.45.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.37.1"},{"key":10,"val":"00000000004.81.1"}]} 2021-03-02T12:28:31.511 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:28:31.512 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:31.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:31.513 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:31.558 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:31.558 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:31.770 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.104.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.57.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.102.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.86.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.45.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.81.1"}}}] 2021-03-02T12:28:31.770 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:28:31.771 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:31.771 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:31.772 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:31.816 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:31.817 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:32.061 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.104.5"},{"key":1,"val":"00000000004.55.1"},{"key":2,"val":"00000000006.57.5"},{"key":3,"val":"00000000004.102.1"},{"key":4,"val":"00000000006.84.5"},{"key":5,"val":"00000000004.80.1"},{"key":6,"val":"00000000004.86.1"},{"key":7,"val":"00000000006.45.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.37.1"},{"key":10,"val":"00000000004.81.1"}]} 2021-03-02T12:28:32.062 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:28:32.062 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:32.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:32.064 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:32.146 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:32.147 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:32.399 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.104.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.57.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.102.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.86.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.45.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.81.1"}}}] 2021-03-02T12:28:32.400 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:28:32.400 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:32.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:32.401 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:32.485 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:32.593 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:32.720 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000006.104.5"},{"key":1,"val":"00000000004.55.1"},{"key":2,"val":"00000000006.57.5"},{"key":3,"val":"00000000004.102.1"},{"key":4,"val":"00000000006.84.5"},{"key":5,"val":"00000000004.80.1"},{"key":6,"val":"00000000004.86.1"},{"key":7,"val":"00000000006.45.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.37.1"},{"key":10,"val":"00000000004.81.1"}]} 2021-03-02T12:28:32.721 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:28:32.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:32.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:32.723 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:32.763 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:32.763 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:33.030 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.104.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.57.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.102.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.80.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.86.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.45.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.81.1"}}}] 2021-03-02T12:28:33.075 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-64 2021-03-02T12:28:33.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:33.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:33.077 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:33.125 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:33.529 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:33.531 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000004.102.5"},{"key":1,"val":"00000000002.49.1"},{"key":2,"val":"00000000004.44.5"},{"key":3,"val":"00000000002.96.1"},{"key":4,"val":"00000000004.88.5"},{"key":5,"val":"00000000002.68.1"},{"key":6,"val":"00000000002.90.1"},{"key":7,"val":"00000000004.44.5"},{"key":8,"val":"00000000002.41.1"},{"key":9,"val":"00000000002.24.1"},{"key":10,"val":"00000000002.69.1"}]} 2021-03-02T12:28:33.531 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:28:33.532 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:33.532 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:33.533 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:33.578 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:33.579 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:33.827 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.102.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.44.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.96.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.88.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.90.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.44.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.24.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}}] 2021-03-02T12:28:33.856 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:28:33.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:33.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:33.858 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:33.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:33.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:34.126 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":"00000000004.102.5"},{"key":1,"val":"00000000002.49.1"},{"key":2,"val":"00000000004.44.5"},{"key":3,"val":"00000000002.96.1"},{"key":4,"val":"00000000004.88.5"},{"key":5,"val":"00000000002.68.1"},{"key":6,"val":"00000000002.90.1"},{"key":7,"val":"00000000004.44.5"},{"key":8,"val":"00000000002.41.1"},{"key":9,"val":"00000000002.24.1"},{"key":10,"val":"00000000002.69.1"}]} 2021-03-02T12:28:34.157 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:28:34.158 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:34.158 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:34.159 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:34.206 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:34.448 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.102.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.49.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.44.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.96.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.88.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.90.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000004.44.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.24.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.1"}}}] 2021-03-02T12:28:34.470 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-64 2021-03-02T12:28:34.471 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:34.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:34.472 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:34.532 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:34.797 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:34.799 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000004.105.1"},{"key":1,"val":"00000000002.50.1"},{"key":2,"val":"00000000004.45.1"},{"key":3,"val":"00000000002.106.1"},{"key":4,"val":"00000000004.93.1"},{"key":5,"val":"00000000002.67.1"},{"key":6,"val":"00000000002.83.1"},{"key":7,"val":"00000000004.40.1"},{"key":8,"val":"00000000002.41.1"},{"key":9,"val":"00000000002.28.1"},{"key":10,"val":"00000000002.68.1"}]} 2021-03-02T12:28:34.800 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:28:34.800 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:34.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:34.801 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:34.887 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:34.887 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:35.131 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.105.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.106.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.93.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.40.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.28.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}}] 2021-03-02T12:28:35.132 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:28:35.132 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:35.133 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:35.133 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:35.191 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:35.191 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:35.414 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":"00000000004.105.1"},{"key":1,"val":"00000000002.50.1"},{"key":2,"val":"00000000004.45.1"},{"key":3,"val":"00000000002.106.1"},{"key":4,"val":"00000000004.93.1"},{"key":5,"val":"00000000002.67.1"},{"key":6,"val":"00000000002.83.1"},{"key":7,"val":"00000000004.40.1"},{"key":8,"val":"00000000002.41.1"},{"key":9,"val":"00000000002.28.1"},{"key":10,"val":"00000000002.68.1"}]} 2021-03-02T12:28:35.415 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:28:35.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:35.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:35.416 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:35.466 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:35.467 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:35.700 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.105.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.50.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.45.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.106.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.93.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.40.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.41.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.28.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.68.1"}}}] 2021-03-02T12:28:35.701 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-64 2021-03-02T12:28:35.702 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-64 zones={test-zone2, test-zone1} 2021-03-02T12:28:35.719 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:35.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:35.729 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:35.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:35.730 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:35.731 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:35.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:35.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:35.732 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:35.733 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:35.743 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:35.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:35.744 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:35.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:35.745 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:35.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:35.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:35.746 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:35.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:35.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:36.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:36.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:36.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:36.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:36.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:36.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:36.719 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:36.844 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:36.909 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:37.038 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:37.142 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:37.194 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:37.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:37.420 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:37.475 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:37.592 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:37.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:37.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:37.876 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:37.987 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:38.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:38.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:38.276 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-64 zones={test-zone2, test-zone1} 2021-03-02T12:28:38.277 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-64 zones={test-zone3, test-zone1} 2021-03-02T12:28:38.286 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:38.292 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:38.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:38.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:38.293 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:38.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:38.294 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:38.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:38.295 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:38.296 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:38.303 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:38.304 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:38.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:38.305 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:38.306 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:38.307 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:38.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:38.308 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:38.324 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:38.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:38.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:38.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:38.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:38.771 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:38.875 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:38.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:39.044 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:39.159 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:39.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:39.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:39.448 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:39.504 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:39.624 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:39.741 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:39.795 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:39.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:40.022 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:40.078 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:40.189 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:40.300 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:40.354 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:40.467 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:40.580 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-64 zones={test-zone3, test-zone1} 2021-03-02T12:28:40.581 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-64 zones={test-zone3, test-zone2} 2021-03-02T12:28:40.593 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:40.601 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:40.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:40.602 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:40.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:40.603 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:40.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:40.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:40.604 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:40.605 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:40.612 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-02T12:28:40.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-02T12:28:40.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-02T12:28:40.613 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-02T12:28:40.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-02T12:28:40.614 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-02T12:28:40.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-02T12:28:40.615 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-02T12:28:40.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:40.687 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:40.803 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-02T12:28:40.910 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:40.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:41.083 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-02T12:28:41.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:41.245 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:41.360 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-02T12:28:41.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:41.539 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:41.653 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-02T12:28:41.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:41.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:41.926 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-02T12:28:42.048 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:42.217 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:42.326 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-02T12:28:42.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:42.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:42.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-02T12:28:42.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:42.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:42.902 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-02T12:28:43.011 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-64 zones={test-zone3, test-zone2} 2021-03-02T12:28:43.195 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:43.196 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:43.197 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:43.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:43.236 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:43.486 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687950.646007_493.1","last_update":"2021-03-02T12:25:50.646007Z"},{"marker":"1_1614687936.963042_722.3","last_update":"2021-03-02T12:25:36.963042Z"},{"marker":"1_1614687950.656932_503.1","last_update":"2021-03-02T12:25:50.656932Z"},{"marker":"1_1614687950.676212_598.1","last_update":"2021-03-02T12:25:50.676212Z"}] 2021-03-02T12:28:43.616 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:28:43.616 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:43.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:43.617 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:43.662 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:43.662 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:43.897 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9531384187116763430},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687950.646007_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.646007Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687936.963042_722.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:36.963042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687950.656932_503.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.656932Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687950.676212_598.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.676212Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:43.897 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-02T12:28:43.898 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:43.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:43.899 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:43.942 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:44.030 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:44.199 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687950.646007_493.1","last_update":"2021-03-02T12:25:50.646007Z"},{"marker":"1_1614687936.963042_722.3","last_update":"2021-03-02T12:25:36.963042Z"},{"marker":"1_1614687950.656932_503.1","last_update":"2021-03-02T12:25:50.656932Z"},{"marker":"1_1614687950.676212_598.1","last_update":"2021-03-02T12:25:50.676212Z"}] 2021-03-02T12:28:44.199 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:28:44.200 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:44.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:44.201 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:44.289 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:44.290 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:44.534 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":7196637147885404157},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687950.646007_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.646007Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687936.963042_722.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:36.963042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687950.656932_503.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.656932Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687950.676212_598.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.676212Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:44.535 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-02T12:28:44.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:44.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:44.536 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:44.626 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:44.626 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:44.855 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687950.646007_493.1","last_update":"2021-03-02T12:25:50.646007Z"},{"marker":"1_1614687936.963042_722.3","last_update":"2021-03-02T12:25:36.963042Z"},{"marker":"1_1614687950.656932_503.1","last_update":"2021-03-02T12:25:50.656932Z"},{"marker":"1_1614687950.676212_598.1","last_update":"2021-03-02T12:25:50.676212Z"}] 2021-03-02T12:28:44.855 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:28:44.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:44.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:44.857 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:44.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:44.901 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:45.164 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9840107722673711902},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687950.646007_493.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.646007Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687936.963042_722.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:36.963042Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687950.656932_503.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.656932Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687950.676212_598.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:50.676212Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:45.165 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone4 source_zone=test-zone1 2021-03-02T12:28:45.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:45.166 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:45.167 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:45.253 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:45.253 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:45.469 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687951.048322_376.9","last_update":"2021-03-02T12:25:51.048322Z"},{"marker":"1_1614687951.048322_716.9","last_update":"2021-03-02T12:25:51.048322Z"},{"marker":"1_1614687951.048322_514.9","last_update":"2021-03-02T12:25:51.048322Z"},{"marker":"1_1614687951.048322_434.6","last_update":"2021-03-02T12:25:51.048322Z"}] 2021-03-02T12:28:45.533 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:28:45.534 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:45.534 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:45.535 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:45.582 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:45.583 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:45.838 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":4819318098326192915},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_376.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_716.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_514.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_434.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:45.840 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-02T12:28:45.841 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:45.845 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:45.847 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:45.929 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:45.929 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:46.156 INFO:teuthology.orchestra.run.smithi036.stdout:[{"marker":"1_1614687951.048322_376.9","last_update":"2021-03-02T12:25:51.048322Z"},{"marker":"1_1614687951.048322_716.9","last_update":"2021-03-02T12:25:51.048322Z"},{"marker":"1_1614687951.048322_514.9","last_update":"2021-03-02T12:25:51.048322Z"},{"marker":"1_1614687951.048322_434.6","last_update":"2021-03-02T12:25:51.048322Z"}] 2021-03-02T12:28:46.156 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:28:46.157 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:46.157 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:46.158 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:46.203 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:46.442 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":13674358676509122085},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_376.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_716.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_514.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687951.048322_434.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:25:51.048322Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:46.443 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone2 2021-03-02T12:28:46.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:46.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:46.445 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:46.536 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:46.780 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687971.348403_231.9","last_update":"2021-03-02T12:26:11.348403Z"},{"marker":"1_1614687971.348403_648.9","last_update":"2021-03-02T12:26:11.348403Z"},{"marker":"1_1614687971.348403_463.9","last_update":"2021-03-02T12:26:11.348403Z"},{"marker":"1_1614687971.348403_312.6","last_update":"2021-03-02T12:26:11.348403Z"}] 2021-03-02T12:28:46.780 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:28:46.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:46.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:46.782 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:46.869 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:46.870 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:47.117 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14630138824005222133},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_231.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_648.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_463.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_312.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:47.118 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone3 2021-03-02T12:28:47.118 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:47.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:47.119 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:47.204 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:47.205 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:47.437 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"1_1614687971.348403_231.9","last_update":"2021-03-02T12:26:11.348403Z"},{"marker":"1_1614687971.348403_648.9","last_update":"2021-03-02T12:26:11.348403Z"},{"marker":"1_1614687971.348403_463.9","last_update":"2021-03-02T12:26:11.348403Z"},{"marker":"1_1614687971.348403_312.6","last_update":"2021-03-02T12:26:11.348403Z"}] 2021-03-02T12:28:47.437 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:28:47.438 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:47.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:47.439 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:47.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:47.486 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:47.700 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":9850944892582908325},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_231.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_648.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_463.9","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1614687971.348403_312.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:26:11.348403Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-02T12:28:47.701 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone3 2021-03-02T12:28:47.702 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2021-03-02T12:28:47.705 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-65 2021-03-02T12:28:47.753 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=majrnq-66 2021-03-02T12:28:47.829 INFO:rgw_multi.tests:create bucket zone=test-zone3 name=majrnq-67 2021-03-02T12:28:49.575 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:49.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:49.577 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:49.620 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:49.880 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.70.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:49.880 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-65 2021-03-02T12:28:49.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:49.881 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:49.882 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:49.925 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:49.926 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:50.149 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:50.149 WARNING:rgw_multi.tests:bucket majrnq-65 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.70.5 target= 2021-03-02T12:28:55.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:55.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:55.152 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:55.198 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:55.199 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:55.444 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.70.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:55.445 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-65 2021-03-02T12:28:55.446 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:55.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:55.447 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:55.489 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:55.716 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.70.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:55.727 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-65 2021-03-02T12:28:55.727 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:55.728 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:55.728 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:55.805 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:55.806 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:56.054 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.70.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:56.090 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-65 2021-03-02T12:28:56.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:56.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:56.092 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:56.143 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:56.144 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:56.380 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.70.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:56.381 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-65 2021-03-02T12:28:56.382 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:56.382 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:56.383 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:56.441 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:56.442 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:56.710 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.70.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:56.711 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-65 2021-03-02T12:28:56.712 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:56.712 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:56.713 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:56.798 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:56.798 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:57.029 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.62.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:57.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-65 2021-03-02T12:28:57.031 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.032 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:57.080 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:57.336 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:57.337 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-65 2021-03-02T12:28:57.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.339 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:57.390 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:57.390 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:57.614 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.62.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:57.616 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-65 2021-03-02T12:28:57.621 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.623 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:57.680 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:57.680 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:57.932 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.62.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:57.933 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-65 2021-03-02T12:28:57.934 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:57.935 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:58.024 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:58.025 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:58.262 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:58.263 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-65 2021-03-02T12:28:58.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:58.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:58.265 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:58.353 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:58.353 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:58.614 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:58.615 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-65 2021-03-02T12:28:58.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:58.616 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:58.617 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:58.700 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:58.934 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:28:58.935 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-65 2021-03-02T12:28:58.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:58.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:28:58.937 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:28:58.981 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:28:58.981 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:28:59.222 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:28:59.222 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-65 2021-03-02T12:28:59.223 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-65 zones={test-zone2, test-zone1} 2021-03-02T12:28:59.233 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:28:59.283 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-02T12:28:59.283 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:28:59.292 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-02T12:28:59.351 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:28:59.590 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:28:59.901 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:00.167 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-65 zones={test-zone2, test-zone1} 2021-03-02T12:29:00.167 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-65 zones={test-zone3, test-zone1} 2021-03-02T12:29:00.176 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:00.223 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-02T12:29:00.223 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:00.230 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-02T12:29:00.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:00.514 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:00.841 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:01.163 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-65 zones={test-zone3, test-zone1} 2021-03-02T12:29:01.164 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-65 zones={test-zone3, test-zone2} 2021-03-02T12:29:01.182 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:01.235 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-02T12:29:01.235 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:01.242 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-02T12:29:01.301 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:01.560 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:01.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-02T12:29:02.227 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-65 zones={test-zone3, test-zone2} 2021-03-02T12:29:02.228 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_multipart_object_sync ... ok 2021-03-02T12:29:02.230 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-68 2021-03-02T12:29:02.363 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:02.366 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:02.367 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:02.407 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:02.408 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:02.606 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:29:02.634 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614688142.265949_640.1","last_update":"2021-03-02T12:29:02.265949Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:29:02.635 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:29:02.636 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:02.637 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:02.638 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:02.680 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:02.681 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:02.909 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688142.254212_632.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:02.254212Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:02.910 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614688142.265949_640.1 target=1_1614688142.254212_632.1 2021-03-02T12:29:03.333 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:03.326+0000 7f79c3fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.348 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:29:03.326+0000 7fb916ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.349 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:03.326+0000 7f79bafed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.350 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:29:03.326+0000 7fb91f7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.350 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:03.330+0000 7f79cb7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.351 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:03.330+0000 7f79ba7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.351 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:29:03.330+0000 7fb917fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.351 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:29:03.330+0000 7fb917fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.355 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:03.328+0000 7fefa6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.355 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:03.332+0000 7fef977fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.356 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:03.332+0000 7fef9cff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:03.356 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:03.332+0000 7fef90ff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:07.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:07.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:07.927 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:07.977 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:07.978 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:08.215 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688142.265949_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:02.265949Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:08.215 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:29:08.216 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:29:08.216 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:08.217 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:08.217 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:08.263 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:08.263 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:08.495 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688142.265949_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:02.265949Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:08.496 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:29:08.496 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:29:08.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:08.499 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:08.500 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:08.545 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:08.545 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:08.811 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688142.265949_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:02.265949Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:08.812 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:29:08.813 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:08.813 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-68', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:08.814 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-68 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:08.862 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:08.862 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:09.095 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.217.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.79.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-02T12:29:09.095 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-68 2021-03-02T12:29:09.096 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:09.096 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-68', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:09.097 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-68 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:09.140 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:09.141 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:09.387 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.217.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.79.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-02T12:29:09.388 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-68 2021-03-02T12:29:09.495 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_encrypted_object_sync ... ok 2021-03-02T12:29:09.497 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-69 2021-03-02T12:29:09.739 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:09.740 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:09.741 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:09.781 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:09.782 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:09.983 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:29:10.016 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614688149.534547_653.1","last_update":"2021-03-02T12:29:09.534547Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:29:10.017 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:29:10.017 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:10.018 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:10.019 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:10.060 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:10.060 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:10.131 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:10.128+0000 7fefa5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:10.132 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:29:10.126+0000 7fb9117f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:10.133 INFO:tasks.rgw.c2.client.0.smithi036.stdout:2021-03-02T12:29:10.126+0000 7fb913ff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:10.133 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:10.128+0000 7fef97fff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:10.134 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:10.126+0000 7f79beff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:10.134 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:10.126+0000 7f79c17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:10.284 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688142.265949_640.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:02.265949Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:10.284 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=1 master=1_1614688149.534547_653.1 target=1_1614688142.265949_640.1 2021-03-02T12:29:15.286 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:15.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:15.287 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:15.328 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:15.328 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:15.541 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688149.534547_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:09.534547Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:15.633 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:29:15.634 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:29:15.635 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:15.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:15.637 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:15.674 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:15.675 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:15.903 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688149.534547_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:09.534547Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:16.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:29:16.039 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:29:16.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:16.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:16.040 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:16.080 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:16.081 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:16.347 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688149.534547_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:09.534547Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:16.422 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:29:16.423 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:16.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:16.425 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:16.466 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:16.467 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:16.697 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.92.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.58.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.60.5"}]} 2021-03-02T12:29:16.754 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-69 2021-03-02T12:29:16.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:16.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:16.758 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:16.802 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:16.802 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:17.039 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}}] 2021-03-02T12:29:17.197 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-69 2021-03-02T12:29:17.198 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:17.198 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:17.199 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:17.242 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:17.478 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.92.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.58.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.60.5"}]} 2021-03-02T12:29:17.489 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-69 2021-03-02T12:29:17.490 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:17.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:17.491 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:17.566 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:17.567 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:17.809 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.58.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}}] 2021-03-02T12:29:17.810 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-69 2021-03-02T12:29:17.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:17.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:17.811 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:17.909 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:17.910 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:18.138 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.92.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.58.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.60.5"}]} 2021-03-02T12:29:18.139 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-69 2021-03-02T12:29:18.139 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:18.140 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:18.140 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:18.201 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:18.201 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:18.465 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.92.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.58.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}}] 2021-03-02T12:29:18.465 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-69 2021-03-02T12:29:18.466 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:18.467 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:18.467 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:18.556 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:18.557 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:18.780 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.60.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.103.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.61.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2021-03-02T12:29:18.781 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-69 2021-03-02T12:29:18.781 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:18.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:18.783 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:18.836 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:18.836 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:19.094 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2021-03-02T12:29:19.094 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-69 2021-03-02T12:29:19.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.096 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:19.143 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:19.143 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:19.366 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.60.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.103.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.61.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2021-03-02T12:29:19.367 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-69 2021-03-02T12:29:19.368 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.371 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:19.422 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:19.422 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:19.675 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2021-03-02T12:29:19.677 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-69 2021-03-02T12:29:19.678 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.682 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:19.761 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:19.762 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:19.988 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.61.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.93.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.51.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2021-03-02T12:29:19.988 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-69 2021-03-02T12:29:19.989 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.989 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:19.990 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:20.073 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:20.325 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.93.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2021-03-02T12:29:20.326 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-69 2021-03-02T12:29:20.326 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:20.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:20.327 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:20.411 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:20.411 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:20.647 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.61.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.93.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.51.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2021-03-02T12:29:20.648 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-69 2021-03-02T12:29:20.648 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:20.649 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-69', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:20.650 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-69 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:20.697 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:20.697 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:20.913 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.93.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.51.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2021-03-02T12:29:20.914 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-69 2021-03-02T12:29:20.915 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-69 zones={test-zone2, test-zone1} 2021-03-02T12:29:20.926 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:20.932 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:20.933 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:20.933 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:20.934 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:20.934 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:20.984 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:20.985 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:20.985 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:21.002 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:21.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:21.176 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:21.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:21.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:21.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:21.556 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:21.610 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:21.723 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:21.835 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:21.890 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:22.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:22.111 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-69 zones={test-zone2, test-zone1} 2021-03-02T12:29:22.111 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-69 zones={test-zone3, test-zone1} 2021-03-02T12:29:22.121 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:22.127 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:22.128 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:22.128 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:22.129 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:22.135 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:22.135 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:22.136 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:22.137 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:22.150 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:22.203 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:22.311 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:22.428 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:22.483 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:22.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:22.699 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:22.755 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:22.864 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:22.972 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:23.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:23.141 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:23.252 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-69 zones={test-zone3, test-zone1} 2021-03-02T12:29:23.253 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-69 zones={test-zone3, test-zone2} 2021-03-02T12:29:23.263 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:23.269 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:23.270 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:23.270 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:23.271 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:23.271 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:23.323 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:23.323 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:23.324 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:23.324 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:23.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:23.436 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:23.551 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:23.657 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:23.711 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:23.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:23.938 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:23.995 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:24.106 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:24.210 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:24.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:24.370 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:24.473 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-69 zones={test-zone3, test-zone2} 2021-03-02T12:29:24.474 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:29:24.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:29:24.476 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:29:24.513 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:24.514 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:24.748 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=majrnq-70 2021-03-02T12:29:24.983 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:24.985 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:24.986 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:25.026 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:25.027 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:25.242 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:29:25.275 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614688164.783975_670.1","last_update":"2021-03-02T12:29:24.783975Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:29:25.276 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:29:25.277 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:25.277 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:25.278 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:25.319 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:25.320 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:25.524 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688164.783975_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:24.783975Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:25.524 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:29:25.525 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:29:25.525 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:25.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:25.526 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:25.568 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:25.783 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688149.534547_653.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:09.534547Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:25.783 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=1 master=1_1614688164.783975_670.1 target=1_1614688149.534547_653.1 2021-03-02T12:29:25.951 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.946+0000 7f79c8ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.952 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.948+0000 7fef907f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.952 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.948+0000 7fefa5ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.953 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.946+0000 7f79c17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.954 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.946+0000 7f79c37fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.954 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.946+0000 7f79be7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.955 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.948+0000 7fefa57fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.955 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.948+0000 7fef94ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.955 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.946+0000 7f79bfff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.956 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.946+0000 7f79bfff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.957 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.952+0000 7fef8efed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.957 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.952+0000 7fefa6ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.957 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.952+0000 7fefa4ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.958 INFO:tasks.rgw.c1.client.1.smithi012.stdout:2021-03-02T12:29:25.952+0000 7fef9f7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.958 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.950+0000 7f79e17fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:25.958 INFO:tasks.rgw.c2.client.1.smithi036.stdout:2021-03-02T12:29:25.950+0000 7f79bc7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-02T12:29:30.784 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:30.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:30.796 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:30.835 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:31.078 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688164.783975_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:24.783975Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:31.079 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:29:31.080 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:29:31.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.081 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:31.121 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:31.121 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:31.391 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688164.783975_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:24.783975Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:31.392 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone4 2021-03-02T12:29:31.392 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.393 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:31.437 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:31.438 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:31.664 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.92.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.73.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.74.5"}]} 2021-03-02T12:29:31.665 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-70 2021-03-02T12:29:31.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.666 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.666 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:31.711 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:31.712 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:31.951 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}}] 2021-03-02T12:29:31.951 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=majrnq-70 2021-03-02T12:29:31.952 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:31.953 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:31.994 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:31.995 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:32.239 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.92.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.73.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.74.5"}]} 2021-03-02T12:29:32.239 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-70 2021-03-02T12:29:32.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:32.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:32.241 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:32.325 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:32.326 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:32.584 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.92.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}}] 2021-03-02T12:29:32.585 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=majrnq-70 2021-03-02T12:29:32.586 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:32.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:32.587 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:32.666 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:32.667 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:32.923 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.71.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.92.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.73.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.74.5"}]} 2021-03-02T12:29:32.924 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-70 2021-03-02T12:29:32.925 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:32.926 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:32.926 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket majrnq-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:32.972 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:32.972 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:33.247 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.71.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.92.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.73.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.74.5"}}}] 2021-03-02T12:29:33.247 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone4 source_zone=test-zone1 bucket=majrnq-70 2021-03-02T12:29:33.248 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:33.248 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:33.249 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:33.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:33.330 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:33.574 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.69.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.98.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.70.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2021-03-02T12:29:33.575 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-70 2021-03-02T12:29:33.576 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:33.576 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:33.577 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:33.621 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:33.873 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.98.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2021-03-02T12:29:33.874 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=majrnq-70 2021-03-02T12:29:33.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:33.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:33.876 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:33.919 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:33.919 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:34.143 INFO:teuthology.orchestra.run.smithi036.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.69.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.98.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.70.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.64.5"}]} 2021-03-02T12:29:34.143 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-70 2021-03-02T12:29:34.144 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:34.144 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:34.145 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:34.191 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:34.192 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:34.447 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.69.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.98.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}}] 2021-03-02T12:29:34.448 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone2 bucket=majrnq-70 2021-03-02T12:29:34.449 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:34.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:34.450 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:34.493 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:34.494 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:34.722 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.105.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.114.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.103.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.73.5"}]} 2021-03-02T12:29:34.722 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-70 2021-03-02T12:29:34.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:34.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:34.724 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:34.809 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:34.809 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:35.057 INFO:teuthology.orchestra.run.smithi012.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.105.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.114.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}}] 2021-03-02T12:29:35.058 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone3 bucket=majrnq-70 2021-03-02T12:29:35.058 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:35.059 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:35.060 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:35.103 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:35.103 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:35.339 INFO:teuthology.orchestra.run.smithi012.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.105.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.114.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.103.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.73.5"}]} 2021-03-02T12:29:35.340 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-70 2021-03-02T12:29:35.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:35.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone3', '--bucket', 'majrnq-70', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:35.341 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone3 --bucket majrnq-70 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:35.385 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:35.386 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:35.620 INFO:teuthology.orchestra.run.smithi036.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.105.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.114.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.103.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.73.5"}}}] 2021-03-02T12:29:35.621 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone3 bucket=majrnq-70 2021-03-02T12:29:35.621 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-70 zones={test-zone2, test-zone1} 2021-03-02T12:29:35.631 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:35.638 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:35.639 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:35.639 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:35.639 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:35.640 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:35.691 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:35.691 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:35.692 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:35.692 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:35.707 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:35.763 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:35.877 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:35.983 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:36.045 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:36.160 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:36.267 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:36.327 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:36.441 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:36.547 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:36.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:36.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:36.907 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-70 zones={test-zone2, test-zone1} 2021-03-02T12:29:36.907 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-70 zones={test-zone3, test-zone1} 2021-03-02T12:29:36.919 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:36.924 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:36.925 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:36.925 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:36.925 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:36.926 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:36.933 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:36.933 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:36.934 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:36.934 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:36.945 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:37.001 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:37.174 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:37.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:37.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:37.465 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:37.571 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:37.626 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:37.744 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:37.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:37.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:38.026 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:38.206 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-70 zones={test-zone3, test-zone1} 2021-03-02T12:29:38.206 INFO:tasks.rgw_multi.zone_rados:comparing bucket=majrnq-70 zones={test-zone3, test-zone2} 2021-03-02T12:29:38.216 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-02T12:29:38.223 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:38.223 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:38.224 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:38.224 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:38.224 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-02T12:29:38.230 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-02T12:29:38.230 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-02T12:29:38.231 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-02T12:29:38.231 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-02T12:29:38.283 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:38.340 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:38.458 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-02T12:29:38.576 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:38.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:38.754 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-02T12:29:38.857 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:38.911 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:39.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-02T12:29:39.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:39.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:39.374 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-02T12:29:39.500 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=majrnq-70 zones={test-zone3, test-zone2} 2021-03-02T12:29:39.500 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:29:39.501 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:29:39.502 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:29:39.543 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:39.544 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:39.811 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:39.812 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'majrnq-70', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:39.813 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket majrnq-70 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:39.894 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:39.895 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:40.152 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:29:40.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:40.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:40.154 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:40.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:40.235 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:40.480 INFO:teuthology.orchestra.run.smithi012.stdout:[{"op_id":"1#00000000001.71.5","op_tag":"781413a3-49d9-4a3c-9ff5-00db01a70f6a.6239.4411","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-02T12:29:09.544994339Z","ver":{"pool":9,"epoch":5},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-69:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.20"}]},{"op_id":"5#00000000001.92.5","op_tag":"781413a3-49d9-4a3c-9ff5-00db01a70f6a.6239.4417","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-02T12:29:09.699505037Z","ver":{"pool":9,"epoch":6},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-69:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.20"}]},{"op_id":"8#00000000001.58.5","op_tag":"781413a3-49d9-4a3c-9ff5-00db01a70f6a.6239.4415","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-02T12:29:09.650179853Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-69:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.20"}]},{"op_id":"10#00000000001.60.5","op_tag":"781413a3-49d9-4a3c-9ff5-00db01a70f6a.6239.4413","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-02T12:29:09.597375321Z","ver":{"pool":9,"epoch":3},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"781413a3-49d9-4a3c-9ff5-00db01a70f6a:majrnq-69:781413a3-49d9-4a3c-9ff5-00db01a70f6a.6241.20"}]}] 2021-03-02T12:29:40.481 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:29:40.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-02T12:29:40.482 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-02T12:29:40.563 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:40.563 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:42.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:42.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'majrnq-69', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:42.150 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket majrnq-69 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:42.229 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:42.230 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:42.490 INFO:teuthology.orchestra.run.smithi012.stdout:[] 2021-03-02T12:29:42.490 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_index_log_trim ... ok 2021-03-02T12:29:42.530 INFO:tasks.rgw_multisite_tests:tasks.rgw_multi.tests.test_bucket_creation_time ... ok 2021-03-02T12:29:42.531 INFO:tasks.rgw_multisite_tests: 2021-03-02T12:29:42.533 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-03-02T12:29:42.533 INFO:tasks.rgw_multisite_tests:Ran 28 tests in 1368.362s 2021-03-02T12:29:42.534 INFO:tasks.rgw_multisite_tests: 2021-03-02T12:29:42.534 INFO:tasks.rgw_multisite_tests:OK 2021-03-02T12:29:42.539 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-03-02T12:29:42.540 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:42.544 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:42.545 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:42.587 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:42.802 INFO:teuthology.orchestra.run.smithi012.stderr:No --period given, using current period=133346b6-56ac-4f8e-91bd-8d51dd761892 2021-03-02T12:29:42.839 INFO:teuthology.orchestra.run.smithi012.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"1_1614688164.783975_670.1","last_update":"2021-03-02T12:29:24.783975Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-02T12:29:42.839 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-02T12:29:42.840 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:42.840 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:42.841 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:42.882 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:42.883 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:43.114 INFO:teuthology.orchestra.run.smithi036.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688164.783975_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:24.783975Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:43.115 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-02T12:29:43.116 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-02T12:29:43.116 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:43.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:43.117 DEBUG:teuthology.orchestra.run.smithi012:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:43.157 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:43.157 INFO:teuthology.orchestra.run.smithi012.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T12:29:43.411 INFO:teuthology.orchestra.run.smithi012.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"133346b6-56ac-4f8e-91bd-8d51dd761892","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1614686849.835843_128.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-02T12:07:29.835843Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"1_1614688164.783975_670.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-02T12:29:24.783975Z","realm_epoch":4}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-02T12:29:43.412 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-02T12:29:43.412 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone4 2021-03-02T12:29:43.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:43.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone4', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-02T12:29:43.414 DEBUG:teuthology.orchestra.run.smithi036:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone4 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-02T12:29:43.455 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setuser ceph since I am not root 2021-03-02T12:29:43.456 INFO:teuthology.orchestra.run.smithi036.stderr:ignoring --setgroup ceph since I am not root 2021-03-02T23:57:39.381 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-02T23:57:40.361 DEBUG:teuthology.task.console_log:Killing console logger for smithi012 2021-03-02T23:57:40.362 DEBUG:teuthology.task.console_log:Killing console logger for smithi036 2021-03-02T23:57:40.363 DEBUG:teuthology.exit:Finished running handlers